Jul 29 22:44:28.878878 ip-10-0-135-98 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 29 22:44:28.878887 ip-10-0-135-98 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 29 22:44:28.878895 ip-10-0-135-98 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 29 22:44:28.879109 ip-10-0-135-98 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 29 22:44:38.893191 ip-10-0-135-98 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 29 22:44:38.893209 ip-10-0-135-98 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d9a1830a000249ba8d57873211e01078 --
Jul 29 22:46:31.803939 ip-10-0-135-98 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 22:46:32.117679 ip-10-0-135-98 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 22:46:32.117679 ip-10-0-135-98 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 22:46:32.117679 ip-10-0-135-98 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 22:46:32.117679 ip-10-0-135-98 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 22:46:32.117679 ip-10-0-135-98 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 22:46:32.119006 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.118923    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 22:46:32.122213 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122198    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:32.122213 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122212    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122217    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122226    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122228    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122232    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122234    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122237    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122239    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122242    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122245    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122247    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122250    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122253    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122256    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122258    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122261    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122263    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122265    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122268    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122271    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:32.122274 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122273    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122276    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122279    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122282    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122285    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122288    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122290    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122293    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122295    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122298    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122300    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122303    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122305    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122308    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122310    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122313    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122316    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122318    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122322    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122326    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:32.122779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122329    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122331    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122334    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122336    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122338    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122341    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122343    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122346    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122348    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122351    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122353    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122356    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122358    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122361    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122365    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122368    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122371    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122373    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122376    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:32.123290 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122379    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122381    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122386    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122389    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122393    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122395    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122399    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122403    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122406    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122409    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122411    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122414    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122416    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122419    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122421    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122425    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122428    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122431    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122434    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:32.123773 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122437    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122440    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122443    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122446    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122448    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122451    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122454    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122457    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122460    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122463    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122466    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122468    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122471    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122473    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122476    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122478    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122481    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122483    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122486    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.122489    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:32.124227 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123202    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123208    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123211    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123215    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123217    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123221    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123224    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123226    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123229    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123232    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123234    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123237    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123240    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123243    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123245    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123248    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123250    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123253    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123256    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123258    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:32.124723 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123261    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123264    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123266    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123269    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123271    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123274    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123277    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123279    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123282    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123284    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123287    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123289    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123292    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123295    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123297    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123300    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123303    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123306    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123309    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123312    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:32.125327 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123314    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123317    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123320    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123322    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123325    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123328    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123330    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123333    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123336    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123340    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123344    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123347    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123349    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123352    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123356    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123359    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123362    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123364    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123367    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123369    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:32.125846 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123372    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123375    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123378    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123381    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123384    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123386    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123389    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123392    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123394    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123398    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123400    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123403    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123406    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123408    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123411    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123414    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123417    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123419    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123422    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123424    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:32.126382 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123427    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123429    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123432    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123435    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123437    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123439    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123442    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123444    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123447    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123449    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123452    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123454    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123457    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123462    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123465    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123468    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123471    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123474    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123477    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:32.126895 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123868    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123873    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123877    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123880    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123883    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123886    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123889    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123891    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123895    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123898    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123901    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123905    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123908    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123911    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123914    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123916    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123919    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123921    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123924    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:32.127356 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123926    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123929    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123931    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123935    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123937    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123940    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123942    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123945    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123947    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123950    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123952    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123955    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123958    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123960    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123963    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123965    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123969    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123972    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123974    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123977    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:32.127833 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123979    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123982    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123985    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123988    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123990    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123993    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123995    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.123998    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124000    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124004    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124008    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124011    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124013    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124018    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124021    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124024    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124027    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124030    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124033    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:32.128332 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124035    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124038    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124041    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124043    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124046    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124049    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124051    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124054    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124059    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124061    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124064    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124067    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124070    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124072    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124075    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124077    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124080    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124082    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124085    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124088    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:32.128815 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124090    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124093    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124095    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124097    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124100    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124103    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124105    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124108    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124110    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124113    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124115    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124118    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124120    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124123    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124125    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124128    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124130    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124133    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124135    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124137    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124141    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:32.129304 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124223    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124232    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124241    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124249    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124254    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124257    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124261    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124265    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124268    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124272    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124275    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124278    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124281    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124284    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124287    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124290    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124293    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124296    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124300    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124303    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124306    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124309    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124313    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124317    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 22:46:32.129824 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124320    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124323    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124327    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124330    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124333    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124336    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124339    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124342    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124346    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124349    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124352    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124355    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124358    2569 flags.go:64] FLAG: --enable-server="true"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124361    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124365    2569 flags.go:64] FLAG: --event-burst="100"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124368    2569 flags.go:64] FLAG: --event-qps="50"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124371    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124373    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124376    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124380    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124383    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124385    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124388    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124391    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124394    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 22:46:32.130417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124397    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124400    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124403    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124406    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124409    2569 flags.go:64] FLAG: --feature-gates=""
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124414    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124417    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124420    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124423    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124426    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124429    2569 flags.go:64] FLAG: --help="false"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124432    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-135-98.ec2.internal"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124435    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124438    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124441    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124445    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124448    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124451    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124454    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124457    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124460    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124463    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124466    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124469    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 29 22:46:32.131096 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124471    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124474    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124477    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124480    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124483    2569 flags.go:64] FLAG: --lock-file=""
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124485    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124488    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124491    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124496    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124499    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124502    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124505    2569 flags.go:64] FLAG: --logging-format="text"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124507    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124512    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124515    2569 flags.go:64] FLAG: --manifest-url=""
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124518    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124522    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124525    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124529    2569 flags.go:64] FLAG: --max-pods="110"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124532    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124535    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124538    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124541    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124544    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124547    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 22:46:32.131779 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124550    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124557    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124560    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124564    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124567    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124570    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124575    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124579    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124582    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124585    2569 flags.go:64] FLAG: --port="10250"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124588    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124591    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02ab3414ed7e0efc6"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124605    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124609    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124612    2569 flags.go:64] FLAG: --register-node="true"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124615    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124619    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124622    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124624    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124627    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124632    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124636    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124639    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124641    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124644    2569 flags.go:64] FLAG: --runonce="false"
Jul 29 22:46:32.132654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124647    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124650    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124653    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124656    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124659    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124662    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124665    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124668    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124671    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124674    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124676    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124680    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124683    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124686    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124689    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124694    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124697    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124699    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124703    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124706    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124709    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124712    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124715    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124717    2569 flags.go:64] FLAG: --v="2"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124721    2569 flags.go:64] FLAG: --version="false"
Jul 29 22:46:32.133289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124725    2569 flags.go:64] FLAG: --vmodule=""
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124729    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.124734    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124848    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124852    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124854    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124859    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124864    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124868    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124871    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124874    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124877    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124880    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124883    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124886    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124889    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124892    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124894    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124897    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124901    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:32.133907 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124905    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124907    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124910    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124913    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124916    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124918    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124921    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124925    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124928    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124931    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124934    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124937    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124939    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124942    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124946    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124948    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124951    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124954    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124956    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124961    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:32.134405 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124964    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124966    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124969    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124971    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124974    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124977    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124979    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124982    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124985    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124987    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124990    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124993    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124995    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.124998    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125001    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125003    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125006    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125009    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125013    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125015    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:32.134918 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125018    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125020    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125023    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125025    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125028    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125031    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125036    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125038    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125041    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125044    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125046    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125051    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125053    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125057    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125059    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125062    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125064    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125067    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125070    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:32.135419 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125072    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125075    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125077    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125080    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125082    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125085    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125088    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125090    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125093    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125095    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125098    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125100    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125103    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125106    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125108    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125111    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125113    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125116    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125118    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:32.135939 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125122    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:32.136398 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125124    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:32.136398 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125127    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:32.136398 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.125130    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:32.136398 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.125632    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 22:46:32.136398 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.133824    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 22:46:32.136398 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.133841    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 22:46:32.136398 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.134798    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 22:46:32.136398 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.134806    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 22:46:32.136398 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134867    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:32.136398 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134873    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:32.136398 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134878    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:32.136398 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134881    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:32.136398 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134885    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:32.136398 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134887    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:32.136398 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134890    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134893    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134895    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134898    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134901    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134904    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134906    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134909    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134912    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134914    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134917    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134920    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134923    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134926    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134929    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134932    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134935    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134937    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134940    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134942    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:32.136789 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134945    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134949    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134952    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134955    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134960    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134963    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134966    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134969    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134971    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134974    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134977    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134980    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134982    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134986    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134990    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134993    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134996    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.134999    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135001    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:32.137268 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135004    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135008    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135011    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135013    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135016    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135018    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135021    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135024    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135026    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135029    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135031    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135034    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135037    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135039    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135042    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135045    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135049    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135052    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135056    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:32.137864 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135060    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135063    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135066    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135068    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135071    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135074    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135076    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135079    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135082    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135084    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135087    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135090    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135092    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135095    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135097    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135105    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135108    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135110    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135113    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135115    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:32.138340 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135118    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:32.138853 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135121    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:32.138853 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135123    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:32.138853 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135126    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:32.138853 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135128    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:32.138853 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135131    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:32.138853 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135134    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:32.138853 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135136    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:32.138853 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135139    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:32.138853 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135142    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:32.138853 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135145    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:32.138853 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135148    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:32.138853 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135150    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:32.138853 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135153    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:32.138853 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135155    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:32.138853 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.135161    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135276    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135282    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135285    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135289    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135292    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135295    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135298    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135300    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135303    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135305    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135308    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135311    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135314    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135317    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135320    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135322    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135325    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135327    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135330    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:32.139259 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135332    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135335    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135338    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135340    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135343    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135346    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135348    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135351    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135354    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135356    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135359    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135361    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135364    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135366    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135369    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135371    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135374    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135376    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135379    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135381    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:32.139779 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135384    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135386    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135388    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135391    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135394    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135396    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135402    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135405    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135408    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135411    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135414    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135417    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135419    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135422    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135425    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135429    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135433    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135436    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135439    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:32.140267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135443    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135446    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135449    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135452    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135456    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135459    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135462    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135465    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135468    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135470    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135473    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135476    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135479    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135482    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135484    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135487    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135489    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135492    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135495    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135498    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:32.140814 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135501    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135504    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135507    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135509    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135512    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135515    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135518    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135520    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135523    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135526    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135528    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135531    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135533    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135536    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135539    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135541    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135544    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135546    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135549    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135551    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:32.141304 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:32.135554    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:32.141801 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.135559    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 22:46:32.141801 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.135723    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 22:46:32.141801 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.137679    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 22:46:32.141801 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.138780    2569 server.go:1019] "Starting client certificate rotation"
Jul 29 22:46:32.141801 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.138883    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 22:46:32.141801 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.139447    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 22:46:32.141801 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.140838    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 22:46:32.142016 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.142003    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 22:46:32.153690 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.153661    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 29 22:46:32.156092 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.156065    2569 log.go:25] "Validated CRI v1 image API"
Jul 29 22:46:32.156822 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.156808    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 22:46:32.158812 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.158794    2569 fs.go:135] Filesystem UUIDs: map[3d8d1708-68a1-430c-a05a-246754ac0ef6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a63c85d0-b987-4b14-8d44-e7c70d29c8c7:/dev/nvme0n1p4]
Jul 29 22:46:32.158871 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.158812    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 22:46:32.164696 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.164669    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 22:46:32.165466 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.165355    2569 manager.go:216] Machine: {Timestamp:2026-07-29 22:46:32.164128244 +0000 UTC m=+0.266209760 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099476 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c58bdc638afc68eb7e91aea7f8cda SystemUUID:ec2c58bd-c638-afc6-8eb7-e91aea7f8cda BootID:d9a1830a-0002-49ba-8d57-873211e01078 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6b:2a:e1:a5:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6b:2a:e1:a5:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:fe:1d:22:7b:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 22:46:32.165548 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.165467    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 22:46:32.165585 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.165556    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 22:46:32.167874 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.167851    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 22:46:32.168011 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.167878    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-98.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 22:46:32.168465 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.168456    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 22:46:32.168495 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.168468    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 22:46:32.168495 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.168475    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 22:46:32.168545 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.168495    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 22:46:32.169250 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.169240    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 22:46:32.171757 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.171745    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 22:46:32.171882 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.171873    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 22:46:32.171916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.171903    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 22:46:32.171916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.171913    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 22:46:32.171972 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.171924    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 22:46:32.171972 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.171934    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 29 22:46:32.171972 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.171942    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 22:46:32.172316 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.172296    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 22:46:32.172366 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.172320    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 22:46:32.174102 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.174086    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 22:46:32.176221 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.176206    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 22:46:32.176319 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.176245    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 22:46:32.177828 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.177814    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 22:46:32.177921 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.177835    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 22:46:32.177921 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.177844    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 22:46:32.177921 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.177853    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 22:46:32.177921 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.177865    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 22:46:32.177921 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.177874    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 22:46:32.177921 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.177886    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 22:46:32.177921 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.177894    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 22:46:32.177921 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.177916    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 22:46:32.177921 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.177925    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 22:46:32.178191 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.177945    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 22:46:32.178191 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.177955    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 22:46:32.178528 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.178518    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 22:46:32.178575 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.178530    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 22:46:32.181069 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:32.181034    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 22:46:32.181069 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:32.181034    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 22:46:32.181069 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.181058    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 22:46:32.181861 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.181846    2569 server.go:1268] "Started kubelet"
Jul 29 22:46:32.182341 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.181932    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 22:46:32.182341 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.181961    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 22:46:32.182341 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.182046    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 22:46:32.182768 ip-10-0-135-98 systemd[1]: Started Kubernetes Kubelet.
Jul 29 22:46:32.183539 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.183519    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 22:46:32.183768 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.183682    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 22:46:32.186801 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:32.185924    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-98.ec2.internal.18c6e42eda55b0e9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-98.ec2.internal,UID:ip-10-0-135-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-98.ec2.internal,},FirstTimestamp:2026-07-29 22:46:32.181657833 +0000 UTC m=+0.283739352,LastTimestamp:2026-07-29 22:46:32.181657833 +0000 UTC m=+0.283739352,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-98.ec2.internal,}"
Jul 29 22:46:32.187063 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:32.187034    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 22:46:32.189311 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.189296    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lddxs"
Jul 29 22:46:32.190999 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.190985    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 22:46:32.191057 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.191004    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 22:46:32.191110 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:32.191069    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Jul 29 22:46:32.191564 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.191511    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 22:46:32.191671 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.191610    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 22:46:32.191671 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.191629    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 22:46:32.191786 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.191730    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 22:46:32.191786 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.191740    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 22:46:32.191912 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.191894    2569 factory.go:55] Registering systemd factory
Jul 29 22:46:32.191912 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.191914    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 29 22:46:32.192208 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.192184    2569 factory.go:153] Registering CRI-O factory
Jul 29 22:46:32.192208 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.192210    2569 factory.go:223] Registration of the crio container factory successfully
Jul 29 22:46:32.192333 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.192272    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 22:46:32.192333 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.192296    2569 factory.go:103] Registering Raw factory
Jul 29 22:46:32.192333 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.192311    2569 manager.go:1196] Started watching for new ooms in manager
Jul 29 22:46:32.193243 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.192984    2569 manager.go:349] Starting recovery of all containers
Jul 29 22:46:32.198539 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.198511    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lddxs"
Jul 29 22:46:32.199388 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:32.199359    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 22:46:32.199388 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:32.199378    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 22:46:32.206160 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.206145    2569 manager.go:354] Recovery completed
Jul 29 22:46:32.209983 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.209972    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:32.211785 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.211768    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:32.211859 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.211798    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:32.211859 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.211813    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:32.212378 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.212362    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 22:46:32.212378 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.212378    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 22:46:32.212489 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.212394    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 22:46:32.212489 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.212370    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:32.213088 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.213074    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:32.213162 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.213098    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:32.213162 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.213108    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:32.213824 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:32.213751    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-98.ec2.internal.18c6e42edc2167e7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-98.ec2.internal,UID:ip-10-0-135-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-98.ec2.internal,},FirstTimestamp:2026-07-29 22:46:32.211785703 +0000 UTC m=+0.313867226,LastTimestamp:2026-07-29 22:46:32.211785703 +0000 UTC m=+0.313867226,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-98.ec2.internal,}"
Jul 29 22:46:32.214918 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.214905    2569 policy_none.go:49] "None policy: Start"
Jul 29 22:46:32.214983 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.214923    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 22:46:32.214983 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.214936    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 22:46:32.216896 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.216883    2569 policy_none.go:47] "Start"
Jul 29 22:46:32.261147 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.261130    2569 manager.go:337] "Starting Device Plugin manager"
Jul 29 22:46:32.261256 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:32.261166    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 22:46:32.261256 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.261180    2569 server.go:87] "Starting device plugin registration server"
Jul 29 22:46:32.261447 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.261434    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 22:46:32.261491 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.261448    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 22:46:32.261560 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.261543    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 22:46:32.261665 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.261641    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 22:46:32.261665 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.261652    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 22:46:32.262138 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:32.262105    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 22:46:32.262212 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:32.262157    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-98.ec2.internal\" not found"
Jul 29 22:46:32.362031 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.361997    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:32.364123 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.362907    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:32.364123 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.362937    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:32.364123 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.362947    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:32.364123 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.362978    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-98.ec2.internal"
Jul 29 22:46:32.368891 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.368810    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-98.ec2.internal"
Jul 29 22:46:32.368891 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:32.368834    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-98.ec2.internal\": node \"ip-10-0-135-98.ec2.internal\" not found"
Jul 29 22:46:32.370149 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.370126    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 22:46:32.371361 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.371342    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 22:46:32.371435 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.371390    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 22:46:32.371435 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.371415    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 22:46:32.371503 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:32.371465    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 22:46:32.373551 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:32.373535    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 22:46:33.481009 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:33.480994    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Jul 29 22:46:33.568990 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.568957    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"]
Jul 29 22:46:33.569091 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.569045    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:33.570929 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.570914    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:33.570991 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.570944    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:33.570991 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.570954    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:33.573352 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.573338    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:33.573470 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.573457    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Jul 29 22:46:33.573524 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.573503    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:33.574081 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.574066    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:33.574147 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.574091    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:33.574147 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.574110    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:33.574147 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.574069    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:33.574250 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.574162    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:33.574250 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.574175    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:33.576306 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.576290    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Jul 29 22:46:33.576394 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.576322    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:33.576983 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.576970    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:33.577051 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.576992    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:33.577051 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.577005    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:33.581253 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:33.581238    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Jul 29 22:46:33.589572 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.589555    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21d832289f76f2c2f5062adee59e8cd8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"21d832289f76f2c2f5062adee59e8cd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Jul 29 22:46:33.589657 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.589586    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21d832289f76f2c2f5062adee59e8cd8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"21d832289f76f2c2f5062adee59e8cd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Jul 29 22:46:33.589657 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.589611    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c3fee7606f52b4c69acd2656db749d26-config\") pod \"kube-apiserver-proxy-ip-10-0-135-98.ec2.internal\" (UID: \"c3fee7606f52b4c69acd2656db749d26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Jul 29 22:46:33.604235 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:33.604213    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-98.ec2.internal\" not found" node="ip-10-0-135-98.ec2.internal"
Jul 29 22:46:33.608545 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:33.608531    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-98.ec2.internal\" not found" node="ip-10-0-135-98.ec2.internal"
Jul 29 22:46:33.682231 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:33.682199    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Jul 29 22:46:33.690557 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.690535    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c3fee7606f52b4c69acd2656db749d26-config\") pod \"kube-apiserver-proxy-ip-10-0-135-98.ec2.internal\" (UID: \"c3fee7606f52b4c69acd2656db749d26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Jul 29 22:46:33.690634 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.690554    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c3fee7606f52b4c69acd2656db749d26-config\") pod \"kube-apiserver-proxy-ip-10-0-135-98.ec2.internal\" (UID: \"c3fee7606f52b4c69acd2656db749d26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Jul 29 22:46:33.690634 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.690568    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21d832289f76f2c2f5062adee59e8cd8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"21d832289f76f2c2f5062adee59e8cd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Jul 29 22:46:33.690634 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.690587    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21d832289f76f2c2f5062adee59e8cd8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"21d832289f76f2c2f5062adee59e8cd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Jul 29 22:46:33.690634 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.690600    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21d832289f76f2c2f5062adee59e8cd8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"21d832289f76f2c2f5062adee59e8cd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Jul 29 22:46:33.690634 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.690626    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21d832289f76f2c2f5062adee59e8cd8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"21d832289f76f2c2f5062adee59e8cd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Jul 29 22:46:33.783024 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:33.782940    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Jul 29 22:46:33.883470 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:33.883431    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Jul 29 22:46:33.907911 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.907888    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Jul 29 22:46:33.911269 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:33.911250    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Jul 29 22:46:33.984456 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:33.984271    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Jul 29 22:46:34.085563 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:34.085459    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Jul 29 22:46:34.185979 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:34.185943    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Jul 29 22:46:34.235408 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:34.235380    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 22:46:34.235858 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:34.235558    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 22:46:34.286576 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:34.286546    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Jul 29 22:46:34.301377 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:34.301350    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 22:41:33 +0000 UTC" deadline="2028-02-17 14:47:11.206409643 +0000 UTC"
Jul 29 22:46:34.301377 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:34.301375    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13624h0m36.905037622s"
Jul 29 22:46:34.315050 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:34.315032    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 22:46:34.386704 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:34.386627    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Jul 29 22:46:34.487570 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:34.487533    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Jul 29 22:46:34.509343 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:34.509311    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3fee7606f52b4c69acd2656db749d26.slice/crio-c301f3359e360df3a65794055d6a326ca7ecd60b7040c45222077aae1dcaeeb6 WatchSource:0}: Error finding container c301f3359e360df3a65794055d6a326ca7ecd60b7040c45222077aae1dcaeeb6: Status 404 returned error can't find the container with id c301f3359e360df3a65794055d6a326ca7ecd60b7040c45222077aae1dcaeeb6
Jul 29 22:46:34.515235 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:34.515219    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 22:46:34.588429 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:34.588384    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Jul 29 22:46:34.597312 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:34.597291    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 22:46:34.683024 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:34.682988    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d832289f76f2c2f5062adee59e8cd8.slice/crio-129fbb02350e62251a944f92408d2c51b5770ee6f63d19a4a910adab8448cb24 WatchSource:0}: Error finding container 129fbb02350e62251a944f92408d2c51b5770ee6f63d19a4a910adab8448cb24: Status 404 returned error can't find the container with id 129fbb02350e62251a944f92408d2c51b5770ee6f63d19a4a910adab8448cb24
Jul 29 22:46:34.686231 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:34.686210    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 22:46:34.688668 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:34.688651    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Jul 29 22:46:34.696040 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:34.696022    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 22:46:34.696828 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:34.696815    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Jul 29 22:46:34.706142 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:34.706124    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 22:46:35.269839 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.269805    2569 apiserver.go:52] "Watching apiserver"
Jul 29 22:46:35.275561 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.275532    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 22:46:35.277545 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.277520    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-w9w6m","kube-system/konnectivity-agent-r4lqd","kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr","openshift-cluster-node-tuning-operator/tuned-86ljg","openshift-image-registry/node-ca-r8cns","openshift-multus/multus-additional-cni-plugins-shgn2","openshift-multus/multus-fxc56","openshift-dns/node-resolver-mbdtd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal","openshift-multus/network-metrics-daemon-t5x5r","openshift-network-diagnostics/network-check-target-54qd4","openshift-network-operator/iptables-alerter-gjsxw","openshift-ovn-kubernetes/ovnkube-node-jvw95"]
Jul 29 22:46:35.280633 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.280014    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.282322 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.282224    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gjsxw"
Jul 29 22:46:35.282566 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.282538    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 22:46:35.282889 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.282868    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 22:46:35.282976 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.282920    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 22:46:35.283042 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.282874    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jhgqg\""
Jul 29 22:46:35.283100 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.283059    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 22:46:35.283100 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.283069    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 22:46:35.284560 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.284123    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 22:46:35.286432 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.286258    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 22:46:35.286546 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.286457    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 22:46:35.286715 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.286693    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cwhnn\""
Jul 29 22:46:35.286909 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.286888    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.288983 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.288506    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 22:46:35.288983 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.288581    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.288983 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.288704    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 22:46:35.289588 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.289263    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 22:46:35.289588 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.289321    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m64b9\""
Jul 29 22:46:35.289588 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.289405    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 22:46:35.289588 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.289462    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 22:46:35.289830 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.289698    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 22:46:35.289970 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.289934    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 22:46:35.293506 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.292913    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xltbk\""
Jul 29 22:46:35.293506 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.293390    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 22:46:35.293782 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.293683    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.293947 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.293922    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 22:46:35.295556 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.294772    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 22:46:35.295556 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.294874    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r8cns"
Jul 29 22:46:35.295744 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.295721    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 22:46:35.295861 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.295756    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 22:46:35.295948 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.295927    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qz2v5\""
Jul 29 22:46:35.297058 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.296828    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 22:46:35.297058 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.297026    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t55jd\""
Jul 29 22:46:35.297236 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.297088    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 22:46:35.297508 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.297353    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 22:46:35.297957 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.297939    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbdtd"
Jul 29 22:46:35.299904 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.299881    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 22:46:35.300125 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300108    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5l9vz\""
Jul 29 22:46:35.300219 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300203    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 22:46:35.300441 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300416    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:35.300544 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300523    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-node-log\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.300611 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300559    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-cni-netd\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.300611 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300594    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e4e9b57-17f8-495a-b460-319701b58ba3-ovnkube-script-lib\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.300713 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300618    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-modprobe-d\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.300713 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4htmn\" (UniqueName: \"kubernetes.io/projected/442fac65-cba1-44e4-bd5b-33fc746a4e54-kube-api-access-4htmn\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.300713 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300667    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ab88d84-44ee-463c-986e-43d1da51284b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.300713 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300694    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-run-netns\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.300908 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300719    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-run-openvswitch\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.300908 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300744    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-log-socket\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.300908 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300767    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e4e9b57-17f8-495a-b460-319701b58ba3-ovnkube-config\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.300908 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300792    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.300908 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:35.300520    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:46:35.300908 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300816    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-lib-modules\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.300908 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300881    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-host\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.301244 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300904    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ab88d84-44ee-463c-986e-43d1da51284b-cnibin\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.301244 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300935    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-var-lib-openvswitch\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.301244 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.300990    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-socket-dir\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.301244 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301012    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-sysctl-conf\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.301244 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301063    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/442fac65-cba1-44e4-bd5b-33fc746a4e54-tmp\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.301244 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301089    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ab88d84-44ee-463c-986e-43d1da51284b-os-release\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.301244 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301128    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ab88d84-44ee-463c-986e-43d1da51284b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.301244 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301153    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ab88d84-44ee-463c-986e-43d1da51284b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.301244 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301194    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whbq6\" (UniqueName: \"kubernetes.io/projected/5ab88d84-44ee-463c-986e-43d1da51284b-kube-api-access-whbq6\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.301244 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5db5k\" (UniqueName: \"kubernetes.io/projected/8e491a98-c409-49a0-9510-8ed3d321366d-kube-api-access-5db5k\") pod \"iptables-alerter-gjsxw\" (UID: \"8e491a98-c409-49a0-9510-8ed3d321366d\") " pod="openshift-network-operator/iptables-alerter-gjsxw"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301271    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-run-systemd\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301296    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-etc-openvswitch\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301332    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-kubelet\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301368    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-systemd-units\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301406    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e4e9b57-17f8-495a-b460-319701b58ba3-ovn-node-metrics-cert\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301447    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6f6p\" (UniqueName: \"kubernetes.io/projected/9e4e9b57-17f8-495a-b460-319701b58ba3-kube-api-access-p6f6p\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301473    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301516    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-kubernetes\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-systemd\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-run\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301601    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-run-ovn-kubernetes\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301630    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e4e9b57-17f8-495a-b460-319701b58ba3-env-overrides\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301667    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-registration-dir\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301690    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-sys-fs\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-sysctl-d\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.301745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301738    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ab88d84-44ee-463c-986e-43d1da51284b-system-cni-dir\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.302681 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301767    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ab88d84-44ee-463c-986e-43d1da51284b-cni-binary-copy\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.302681 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301791    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8e491a98-c409-49a0-9510-8ed3d321366d-iptables-alerter-script\") pod \"iptables-alerter-gjsxw\" (UID: \"8e491a98-c409-49a0-9510-8ed3d321366d\") " pod="openshift-network-operator/iptables-alerter-gjsxw"
Jul 29 22:46:35.302681 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301814    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e491a98-c409-49a0-9510-8ed3d321366d-host-slash\") pod \"iptables-alerter-gjsxw\" (UID: \"8e491a98-c409-49a0-9510-8ed3d321366d\") " pod="openshift-network-operator/iptables-alerter-gjsxw"
Jul 29 22:46:35.302681 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301884    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-cni-bin\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.302681 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301917    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.302681 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301945    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-device-dir\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.302681 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.301980    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcwcm\" (UniqueName: \"kubernetes.io/projected/9d48fa5a-4721-40b8-983e-c6febef5ea3b-kube-api-access-tcwcm\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.302681 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.302011    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-sysconfig\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.302681 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.302037    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-sys\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.302681 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.302062    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-var-lib-kubelet\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.302681 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.302079    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-tuned\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.302681 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.302099    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-slash\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.302681 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.302124    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-run-ovn\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.303678 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.303657    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r4lqd"
Jul 29 22:46:35.305505 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.305471    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.305914 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.305894    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 22:46:35.306012 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.305927    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vq7gx\""
Jul 29 22:46:35.306012 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.305970    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 22:46:35.306012 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.305970    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 22:46:35.307208 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.307181    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 22:46:35.307380 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.307363    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7fwlw\""
Jul 29 22:46:35.308202 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.308182    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:35.308287 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:35.308259    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:46:35.311631 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.311609    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:35.311753 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:35.311701    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:46:35.328116 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.328087    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xtnd6"
Jul 29 22:46:35.336554 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.336527    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xtnd6"
Jul 29 22:46:35.390019 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.389996    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 22:46:35.402513 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.402451    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-run-netns\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.402666 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.402537    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-run-netns\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.402666 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.402589    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-run-openvswitch\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.402666 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.402644    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-run-openvswitch\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.402831 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.402665    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e4e9b57-17f8-495a-b460-319701b58ba3-ovnkube-config\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.402831 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.402702    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:35.402831 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.402725    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-multus-cni-dir\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.402831 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.402776    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-socket-dir\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.402831 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.402804    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-sysctl-conf\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.403099 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.402848    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/450449c0-a316-47ca-81cd-844420f36bea-kubelet-config\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:35.403099 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.402997    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-sysctl-conf\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.403099 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403043    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-socket-dir\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.403099 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403084    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ab88d84-44ee-463c-986e-43d1da51284b-os-release\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.403300 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403110    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ab88d84-44ee-463c-986e-43d1da51284b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.403300 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403135    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whbq6\" (UniqueName: \"kubernetes.io/projected/5ab88d84-44ee-463c-986e-43d1da51284b-kube-api-access-whbq6\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.403300 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403161    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-run-systemd\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.403300 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403183    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-etc-openvswitch\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.403300 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403206    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6f6p\" (UniqueName: \"kubernetes.io/projected/9e4e9b57-17f8-495a-b460-319701b58ba3-kube-api-access-p6f6p\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.403300 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403229    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-kubernetes\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.403300 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403252    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e4e9b57-17f8-495a-b460-319701b58ba3-ovn-node-metrics-cert\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.403300 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403275    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-systemd\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.403300 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403282    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ab88d84-44ee-463c-986e-43d1da51284b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.403300 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403297    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-lib-modules\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.403712 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403351    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e4e9b57-17f8-495a-b460-319701b58ba3-ovnkube-config\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.403712 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403390    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ab88d84-44ee-463c-986e-43d1da51284b-os-release\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.403712 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403421    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-system-cni-dir\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.403712 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403434    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-lib-modules\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.403712 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403453    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-cnibin\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.403712 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403477    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-var-lib-cni-multus\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.403712 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403506    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-kubernetes\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.404032 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403523    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-run-ovn-kubernetes\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.404032 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403952    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e4e9b57-17f8-495a-b460-319701b58ba3-env-overrides\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.404032 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.403996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-sys-fs\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.405697 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.404294    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-systemd\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.405697 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.404899    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-run-ovn-kubernetes\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.405697 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.404541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-sysctl-d\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.405697 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.405172    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f678c5dc-4755-4702-9be8-79557973646a-agent-certs\") pod \"konnectivity-agent-r4lqd\" (UID: \"f678c5dc-4755-4702-9be8-79557973646a\") " pod="kube-system/konnectivity-agent-r4lqd"
Jul 29 22:46:35.405697 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.405208    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-hostroot\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.405697 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.405244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-cni-bin\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.405697 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.405359    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.405697 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.405435    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.405697 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.405501    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e4e9b57-17f8-495a-b460-319701b58ba3-env-overrides\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.405697 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.405572    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-cni-bin\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.405697 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.405648    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-sys-fs\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.406260 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.405735    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-run-systemd\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.406260 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.405773    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-sysctl-d\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.406260 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.405872    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-etc-openvswitch\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.406260 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.405981    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-device-dir\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.406260 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406041    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-sys\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.406260 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406072    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-var-lib-kubelet\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.406260 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406125    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-run-k8s-cni-cncf-io\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.406260 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406175    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-etc-kubernetes\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.406260 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-tuned\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.406686 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406281    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-var-lib-kubelet\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.406686 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406227    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f404ba-6594-4814-b5c5-9904a205207c-tmp-dir\") pod \"node-resolver-mbdtd\" (UID: \"06f404ba-6594-4814-b5c5-9904a205207c\") " pod="openshift-dns/node-resolver-mbdtd"
Jul 29 22:46:35.406686 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm27s\" (UniqueName: \"kubernetes.io/projected/6289a7d5-1978-462d-9ffc-a4135ff29c6d-kube-api-access-vm27s\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.406686 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406379    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:35.406686 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406400    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-device-dir\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.406686 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406404    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-slash\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.406686 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406461    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-sys\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.406686 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-run-ovn\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.406686 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406606    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-slash\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.406686 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406611    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-node-log\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.407106 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406683    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 22:46:35.407106 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406794    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e4e9b57-17f8-495a-b460-319701b58ba3-ovnkube-script-lib\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.407106 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406838    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlx5m\" (UniqueName: \"kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m\") pod \"network-check-target-54qd4\" (UID: \"2fcc94dd-27b8-4ee6-930c-be6245e627c0\") " pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:35.407106 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.406891    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ab88d84-44ee-463c-986e-43d1da51284b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.407106 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.407046    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-log-socket\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.407336 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.407107    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.407336 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.407153    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzrkm\" (UniqueName: \"kubernetes.io/projected/06f404ba-6594-4814-b5c5-9904a205207c-kube-api-access-kzrkm\") pod \"node-resolver-mbdtd\" (UID: \"06f404ba-6594-4814-b5c5-9904a205207c\") " pod="openshift-dns/node-resolver-mbdtd"
Jul 29 22:46:35.407336 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.407243    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-run-ovn\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.407336 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.407189    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-multus-conf-dir\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.407336 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.407308    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-node-log\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.407336 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.407313    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ab88d84-44ee-463c-986e-43d1da51284b-cnibin\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.407356    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-var-lib-openvswitch\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.407627    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ab88d84-44ee-463c-986e-43d1da51284b-cnibin\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.407761    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.407627    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-var-lib-openvswitch\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.407807    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8402f600-296c-4fa7-a6a2-1d34bb23781c-serviceca\") pod \"node-ca-r8cns\" (UID: \"8402f600-296c-4fa7-a6a2-1d34bb23781c\") " pod="openshift-image-registry/node-ca-r8cns"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.407808    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-log-socket\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.407973    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-var-lib-cni-bin\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408010    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ab88d84-44ee-463c-986e-43d1da51284b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408044    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5db5k\" (UniqueName: \"kubernetes.io/projected/8e491a98-c409-49a0-9510-8ed3d321366d-kube-api-access-5db5k\") pod \"iptables-alerter-gjsxw\" (UID: \"8e491a98-c409-49a0-9510-8ed3d321366d\") " pod="openshift-network-operator/iptables-alerter-gjsxw"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408078    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408179    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ab88d84-44ee-463c-986e-43d1da51284b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408186    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-run\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408214    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-host\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-os-release\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408272    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-run-netns\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408305    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-var-lib-kubelet\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.411181 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408345    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-kubelet\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408379    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-systemd-units\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408435    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408496    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/450449c0-a316-47ca-81cd-844420f36bea-dbus\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408436    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e4e9b57-17f8-495a-b460-319701b58ba3-ovnkube-script-lib\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6289a7d5-1978-462d-9ffc-a4135ff29c6d-multus-daemon-config\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408632    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-run\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408641    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-host\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408718    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-registration-dir\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408769    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-systemd-units\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408786    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ab88d84-44ee-463c-986e-43d1da51284b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408814    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-kubelet\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408850    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8402f600-296c-4fa7-a6a2-1d34bb23781c-host\") pod \"node-ca-r8cns\" (UID: \"8402f600-296c-4fa7-a6a2-1d34bb23781c\") " pod="openshift-image-registry/node-ca-r8cns"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408882    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-multus-socket-dir-parent\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408910    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-run-multus-certs\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408940    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ab88d84-44ee-463c-986e-43d1da51284b-system-cni-dir\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408974    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ab88d84-44ee-463c-986e-43d1da51284b-cni-binary-copy\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.413271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.408982    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d48fa5a-4721-40b8-983e-c6febef5ea3b-registration-dir\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409008    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8e491a98-c409-49a0-9510-8ed3d321366d-iptables-alerter-script\") pod \"iptables-alerter-gjsxw\" (UID: \"8e491a98-c409-49a0-9510-8ed3d321366d\") " pod="openshift-network-operator/iptables-alerter-gjsxw"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409050    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e491a98-c409-49a0-9510-8ed3d321366d-host-slash\") pod \"iptables-alerter-gjsxw\" (UID: \"8e491a98-c409-49a0-9510-8ed3d321366d\") " pod="openshift-network-operator/iptables-alerter-gjsxw"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409082    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcwcm\" (UniqueName: \"kubernetes.io/projected/9d48fa5a-4721-40b8-983e-c6febef5ea3b-kube-api-access-tcwcm\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409117    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-sysconfig\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409163    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwtsg\" (UniqueName: \"kubernetes.io/projected/8402f600-296c-4fa7-a6a2-1d34bb23781c-kube-api-access-hwtsg\") pod \"node-ca-r8cns\" (UID: \"8402f600-296c-4fa7-a6a2-1d34bb23781c\") " pod="openshift-image-registry/node-ca-r8cns"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409186    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06f404ba-6594-4814-b5c5-9904a205207c-hosts-file\") pod \"node-resolver-mbdtd\" (UID: \"06f404ba-6594-4814-b5c5-9904a205207c\") " pod="openshift-dns/node-resolver-mbdtd"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409204    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ab88d84-44ee-463c-986e-43d1da51284b-system-cni-dir\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409249    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6289a7d5-1978-462d-9ffc-a4135ff29c6d-cni-binary-copy\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409280    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-cni-netd\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409336    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-modprobe-d\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409367    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4htmn\" (UniqueName: \"kubernetes.io/projected/442fac65-cba1-44e4-bd5b-33fc746a4e54-kube-api-access-4htmn\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409391    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f678c5dc-4755-4702-9be8-79557973646a-konnectivity-ca\") pod \"konnectivity-agent-r4lqd\" (UID: \"f678c5dc-4755-4702-9be8-79557973646a\") " pod="kube-system/konnectivity-agent-r4lqd"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409436    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbj4d\" (UniqueName: \"kubernetes.io/projected/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-kube-api-access-bbj4d\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409467    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-sysconfig\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409494    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/442fac65-cba1-44e4-bd5b-33fc746a4e54-tmp\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409517    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8e491a98-c409-49a0-9510-8ed3d321366d-iptables-alerter-script\") pod \"iptables-alerter-gjsxw\" (UID: \"8e491a98-c409-49a0-9510-8ed3d321366d\") " pod="openshift-network-operator/iptables-alerter-gjsxw"
Jul 29 22:46:35.414109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409549    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e491a98-c409-49a0-9510-8ed3d321366d-host-slash\") pod \"iptables-alerter-gjsxw\" (UID: \"8e491a98-c409-49a0-9510-8ed3d321366d\") " pod="openshift-network-operator/iptables-alerter-gjsxw"
Jul 29 22:46:35.414949 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.409811    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e4e9b57-17f8-495a-b460-319701b58ba3-host-cni-netd\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.414949 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.410133    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ab88d84-44ee-463c-986e-43d1da51284b-cni-binary-copy\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.414949 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.410204    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-modprobe-d\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.414949 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.413207    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/442fac65-cba1-44e4-bd5b-33fc746a4e54-tmp\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.414949 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.413219    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e4e9b57-17f8-495a-b460-319701b58ba3-ovn-node-metrics-cert\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.414949 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.413848    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whbq6\" (UniqueName: \"kubernetes.io/projected/5ab88d84-44ee-463c-986e-43d1da51284b-kube-api-access-whbq6\") pod \"multus-additional-cni-plugins-shgn2\" (UID: \"5ab88d84-44ee-463c-986e-43d1da51284b\") " pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.414949 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.414175    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/442fac65-cba1-44e4-bd5b-33fc746a4e54-etc-tuned\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.416601 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.416574    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6f6p\" (UniqueName: \"kubernetes.io/projected/9e4e9b57-17f8-495a-b460-319701b58ba3-kube-api-access-p6f6p\") pod \"ovnkube-node-jvw95\" (UID: \"9e4e9b57-17f8-495a-b460-319701b58ba3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.416837 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.416815    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5db5k\" (UniqueName: \"kubernetes.io/projected/8e491a98-c409-49a0-9510-8ed3d321366d-kube-api-access-5db5k\") pod \"iptables-alerter-gjsxw\" (UID: \"8e491a98-c409-49a0-9510-8ed3d321366d\") " pod="openshift-network-operator/iptables-alerter-gjsxw"
Jul 29 22:46:35.417428 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.417385    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcwcm\" (UniqueName: \"kubernetes.io/projected/9d48fa5a-4721-40b8-983e-c6febef5ea3b-kube-api-access-tcwcm\") pod \"aws-ebs-csi-driver-node-47wpr\" (UID: \"9d48fa5a-4721-40b8-983e-c6febef5ea3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.418097 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.418079    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4htmn\" (UniqueName: \"kubernetes.io/projected/442fac65-cba1-44e4-bd5b-33fc746a4e54-kube-api-access-4htmn\") pod \"tuned-86ljg\" (UID: \"442fac65-cba1-44e4-bd5b-33fc746a4e54\") " pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.473470 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.473416    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal" event={"ID":"21d832289f76f2c2f5062adee59e8cd8","Type":"ContainerStarted","Data":"129fbb02350e62251a944f92408d2c51b5770ee6f63d19a4a910adab8448cb24"}
Jul 29 22:46:35.474817 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.474770    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal" event={"ID":"c3fee7606f52b4c69acd2656db749d26","Type":"ContainerStarted","Data":"c301f3359e360df3a65794055d6a326ca7ecd60b7040c45222077aae1dcaeeb6"}
Jul 29 22:46:35.510592 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510559    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzrkm\" (UniqueName: \"kubernetes.io/projected/06f404ba-6594-4814-b5c5-9904a205207c-kube-api-access-kzrkm\") pod \"node-resolver-mbdtd\" (UID: \"06f404ba-6594-4814-b5c5-9904a205207c\") " pod="openshift-dns/node-resolver-mbdtd"
Jul 29 22:46:35.510771 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510598    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-multus-conf-dir\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.510771 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510636    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8402f600-296c-4fa7-a6a2-1d34bb23781c-serviceca\") pod \"node-ca-r8cns\" (UID: \"8402f600-296c-4fa7-a6a2-1d34bb23781c\") " pod="openshift-image-registry/node-ca-r8cns"
Jul 29 22:46:35.510771 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510654    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-var-lib-cni-bin\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.510771 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510683    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-os-release\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.510771 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510705    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-run-netns\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.510771 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510729    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-var-lib-kubelet\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.510771 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510730    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-multus-conf-dir\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.510771 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510752    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/450449c0-a316-47ca-81cd-844420f36bea-dbus\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:35.511126 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510794    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-run-netns\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.511126 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510806    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-var-lib-kubelet\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.511126 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510822    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6289a7d5-1978-462d-9ffc-a4135ff29c6d-multus-daemon-config\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.511126 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510843    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-var-lib-cni-bin\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.511126 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510840    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-os-release\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.511126 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510848    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8402f600-296c-4fa7-a6a2-1d34bb23781c-host\") pod \"node-ca-r8cns\" (UID: \"8402f600-296c-4fa7-a6a2-1d34bb23781c\") " pod="openshift-image-registry/node-ca-r8cns"
Jul 29 22:46:35.511126 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510876    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8402f600-296c-4fa7-a6a2-1d34bb23781c-host\") pod \"node-ca-r8cns\" (UID: \"8402f600-296c-4fa7-a6a2-1d34bb23781c\") " pod="openshift-image-registry/node-ca-r8cns"
Jul 29 22:46:35.511126 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510922    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/450449c0-a316-47ca-81cd-844420f36bea-dbus\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:35.511126 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510981    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-multus-socket-dir-parent\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.511126 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.510924    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-multus-socket-dir-parent\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.511126 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511030    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-run-multus-certs\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.511126 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511065    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwtsg\" (UniqueName: \"kubernetes.io/projected/8402f600-296c-4fa7-a6a2-1d34bb23781c-kube-api-access-hwtsg\") pod \"node-ca-r8cns\" (UID: \"8402f600-296c-4fa7-a6a2-1d34bb23781c\") " pod="openshift-image-registry/node-ca-r8cns"
Jul 29 22:46:35.511126 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511091    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06f404ba-6594-4814-b5c5-9904a205207c-hosts-file\") pod \"node-resolver-mbdtd\" (UID: \"06f404ba-6594-4814-b5c5-9904a205207c\") " pod="openshift-dns/node-resolver-mbdtd"
Jul 29 22:46:35.511126 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511108    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-run-multus-certs\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.511126 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511115    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6289a7d5-1978-462d-9ffc-a4135ff29c6d-cni-binary-copy\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511162    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f678c5dc-4755-4702-9be8-79557973646a-konnectivity-ca\") pod \"konnectivity-agent-r4lqd\" (UID: \"f678c5dc-4755-4702-9be8-79557973646a\") " pod="kube-system/konnectivity-agent-r4lqd"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511065    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511190    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbj4d\" (UniqueName: \"kubernetes.io/projected/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-kube-api-access-bbj4d\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511219    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511245    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-multus-cni-dir\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511273    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/450449c0-a316-47ca-81cd-844420f36bea-kubelet-config\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511307    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-system-cni-dir\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511359    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06f404ba-6594-4814-b5c5-9904a205207c-hosts-file\") pod \"node-resolver-mbdtd\" (UID: \"06f404ba-6594-4814-b5c5-9904a205207c\") " pod="openshift-dns/node-resolver-mbdtd"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511373    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-cnibin\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511425    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-cnibin\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511429    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-var-lib-cni-multus\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511434    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-multus-cni-dir\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511164    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8402f600-296c-4fa7-a6a2-1d34bb23781c-serviceca\") pod \"node-ca-r8cns\" (UID: \"8402f600-296c-4fa7-a6a2-1d34bb23781c\") " pod="openshift-image-registry/node-ca-r8cns"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:35.511429    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511469    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f678c5dc-4755-4702-9be8-79557973646a-agent-certs\") pod \"konnectivity-agent-r4lqd\" (UID: \"f678c5dc-4755-4702-9be8-79557973646a\") " pod="kube-system/konnectivity-agent-r4lqd"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511504    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/450449c0-a316-47ca-81cd-844420f36bea-kubelet-config\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511512    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-hostroot\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512299 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511515    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-system-cni-dir\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:35.511551    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret podName:450449c0-a316-47ca-81cd-844420f36bea nodeName:}" failed. No retries permitted until 2026-07-29 22:46:36.011529209 +0000 UTC m=+3.016726623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret") pod "global-pull-secret-syncer-w9w6m" (UID: "450449c0-a316-47ca-81cd-844420f36bea") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511553    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-hostroot\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511591    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-var-lib-cni-multus\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511620    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-run-k8s-cni-cncf-io\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511644    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-etc-kubernetes\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511667    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f404ba-6594-4814-b5c5-9904a205207c-tmp-dir\") pod \"node-resolver-mbdtd\" (UID: \"06f404ba-6594-4814-b5c5-9904a205207c\") " pod="openshift-dns/node-resolver-mbdtd"
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511691    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm27s\" (UniqueName: \"kubernetes.io/projected/6289a7d5-1978-462d-9ffc-a4135ff29c6d-kube-api-access-vm27s\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511703    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-host-run-k8s-cni-cncf-io\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511716    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511745    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlx5m\" (UniqueName: \"kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m\") pod \"network-check-target-54qd4\" (UID: \"2fcc94dd-27b8-4ee6-930c-be6245e627c0\") " pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511790    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f678c5dc-4755-4702-9be8-79557973646a-konnectivity-ca\") pod \"konnectivity-agent-r4lqd\" (UID: \"f678c5dc-4755-4702-9be8-79557973646a\") " pod="kube-system/konnectivity-agent-r4lqd"
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.511865    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6289a7d5-1978-462d-9ffc-a4135ff29c6d-etc-kubernetes\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:35.511952    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.512004    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6289a7d5-1978-462d-9ffc-a4135ff29c6d-multus-daemon-config\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:35.512025    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs podName:0d19dc00-e3da-4306-a5ff-8cca92e02a1c nodeName:}" failed. No retries permitted until 2026-07-29 22:46:36.012006666 +0000 UTC m=+3.017204072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs") pod "network-metrics-daemon-t5x5r" (UID: "0d19dc00-e3da-4306-a5ff-8cca92e02a1c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.512007    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f404ba-6594-4814-b5c5-9904a205207c-tmp-dir\") pod \"node-resolver-mbdtd\" (UID: \"06f404ba-6594-4814-b5c5-9904a205207c\") " pod="openshift-dns/node-resolver-mbdtd"
Jul 29 22:46:35.512916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.512265    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6289a7d5-1978-462d-9ffc-a4135ff29c6d-cni-binary-copy\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.514137 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.514115    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f678c5dc-4755-4702-9be8-79557973646a-agent-certs\") pod \"konnectivity-agent-r4lqd\" (UID: \"f678c5dc-4755-4702-9be8-79557973646a\") " pod="kube-system/konnectivity-agent-r4lqd"
Jul 29 22:46:35.521022 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:35.520963    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:35.521022 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:35.520993    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:35.521022 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:35.521008    2569 projected.go:196] Error preparing data for projected volume kube-api-access-rlx5m for pod openshift-network-diagnostics/network-check-target-54qd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:35.521229 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:35.521105    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m podName:2fcc94dd-27b8-4ee6-930c-be6245e627c0 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:36.021089355 +0000 UTC m=+3.026286769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rlx5m" (UniqueName: "kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m") pod "network-check-target-54qd4" (UID: "2fcc94dd-27b8-4ee6-930c-be6245e627c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:35.523941 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.523914    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm27s\" (UniqueName: \"kubernetes.io/projected/6289a7d5-1978-462d-9ffc-a4135ff29c6d-kube-api-access-vm27s\") pod \"multus-fxc56\" (UID: \"6289a7d5-1978-462d-9ffc-a4135ff29c6d\") " pod="openshift-multus/multus-fxc56"
Jul 29 22:46:35.524046 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.523923    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwtsg\" (UniqueName: \"kubernetes.io/projected/8402f600-296c-4fa7-a6a2-1d34bb23781c-kube-api-access-hwtsg\") pod \"node-ca-r8cns\" (UID: \"8402f600-296c-4fa7-a6a2-1d34bb23781c\") " pod="openshift-image-registry/node-ca-r8cns"
Jul 29 22:46:35.524144 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.524127    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbj4d\" (UniqueName: \"kubernetes.io/projected/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-kube-api-access-bbj4d\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:35.524250 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.524234    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzrkm\" (UniqueName: \"kubernetes.io/projected/06f404ba-6594-4814-b5c5-9904a205207c-kube-api-access-kzrkm\") pod \"node-resolver-mbdtd\" (UID: \"06f404ba-6594-4814-b5c5-9904a205207c\") " pod="openshift-dns/node-resolver-mbdtd"
Jul 29 22:46:35.597889 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.597853    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-shgn2"
Jul 29 22:46:35.609232 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.609197    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gjsxw"
Jul 29 22:46:35.618324 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.618291    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:46:35.626353 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.626322    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr"
Jul 29 22:46:35.633065 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.633046    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-86ljg"
Jul 29 22:46:35.641790 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.641768    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r8cns"
Jul 29 22:46:35.648437 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.648419    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbdtd"
Jul 29 22:46:35.660059 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.660032    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r4lqd"
Jul 29 22:46:35.673827 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:35.673788    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fxc56"
Jul 29 22:46:36.015027 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.014952    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:36.015027 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.014997    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:36.015191 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:36.015115    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:36.015191 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:36.015175    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret podName:450449c0-a316-47ca-81cd-844420f36bea nodeName:}" failed. No retries permitted until 2026-07-29 22:46:37.015160086 +0000 UTC m=+4.020357491 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret") pod "global-pull-secret-syncer-w9w6m" (UID: "450449c0-a316-47ca-81cd-844420f36bea") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:36.015278 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:36.015115    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:36.015278 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:36.015252    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs podName:0d19dc00-e3da-4306-a5ff-8cca92e02a1c nodeName:}" failed. No retries permitted until 2026-07-29 22:46:37.015224843 +0000 UTC m=+4.020422244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs") pod "network-metrics-daemon-t5x5r" (UID: "0d19dc00-e3da-4306-a5ff-8cca92e02a1c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:36.115957 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.115917    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlx5m\" (UniqueName: \"kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m\") pod \"network-check-target-54qd4\" (UID: \"2fcc94dd-27b8-4ee6-930c-be6245e627c0\") " pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:36.116128 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:36.116110    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:36.116178 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:36.116138    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:36.116178 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:36.116152    2569 projected.go:196] Error preparing data for projected volume kube-api-access-rlx5m for pod openshift-network-diagnostics/network-check-target-54qd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:36.116282 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:36.116212    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m podName:2fcc94dd-27b8-4ee6-930c-be6245e627c0 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:37.116193137 +0000 UTC m=+4.121390542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlx5m" (UniqueName: "kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m") pod "network-check-target-54qd4" (UID: "2fcc94dd-27b8-4ee6-930c-be6245e627c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:36.340774 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.340692    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 22:41:35 +0000 UTC" deadline="2028-03-09 13:08:00.676395738 +0000 UTC"
Jul 29 22:46:36.340774 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.340729    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14126h21m24.335671032s"
Jul 29 22:46:36.384784 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:36.384757    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d48fa5a_4721_40b8_983e_c6febef5ea3b.slice/crio-68375381af9045ed4241105bc95cb4fc224d1cadd89b73538147025638c7728d WatchSource:0}: Error finding container 68375381af9045ed4241105bc95cb4fc224d1cadd89b73538147025638c7728d: Status 404 returned error can't find the container with id 68375381af9045ed4241105bc95cb4fc224d1cadd89b73538147025638c7728d
Jul 29 22:46:36.385985 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:36.385957    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e491a98_c409_49a0_9510_8ed3d321366d.slice/crio-270383b29e6fa3c09d7635f676b77e109c69b115c4c87cf291966eb19dc71bba WatchSource:0}: Error finding container 270383b29e6fa3c09d7635f676b77e109c69b115c4c87cf291966eb19dc71bba: Status 404 returned error can't find the container with id 270383b29e6fa3c09d7635f676b77e109c69b115c4c87cf291966eb19dc71bba
Jul 29 22:46:36.389886 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:36.389804    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442fac65_cba1_44e4_bd5b_33fc746a4e54.slice/crio-84b74b55d0c4ebdd506ddd98d99edde976312a466bedc6374f2b44a565da6192 WatchSource:0}: Error finding container 84b74b55d0c4ebdd506ddd98d99edde976312a466bedc6374f2b44a565da6192: Status 404 returned error can't find the container with id 84b74b55d0c4ebdd506ddd98d99edde976312a466bedc6374f2b44a565da6192
Jul 29 22:46:36.391031 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:36.391012    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4e9b57_17f8_495a_b460_319701b58ba3.slice/crio-99a8250bea8ca81024e3081b8531ee0cf6cca01e1abf531af6543875e1f3a45c WatchSource:0}: Error finding container 99a8250bea8ca81024e3081b8531ee0cf6cca01e1abf531af6543875e1f3a45c: Status 404 returned error can't find the container with id 99a8250bea8ca81024e3081b8531ee0cf6cca01e1abf531af6543875e1f3a45c
Jul 29 22:46:36.391947 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:36.391926    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab88d84_44ee_463c_986e_43d1da51284b.slice/crio-b12a315d4a0d163595dc8146cb75857a616cc2cb3dc9d09ff7ecfea515d5b415 WatchSource:0}: Error finding container b12a315d4a0d163595dc8146cb75857a616cc2cb3dc9d09ff7ecfea515d5b415: Status 404 returned error can't find the container with id b12a315d4a0d163595dc8146cb75857a616cc2cb3dc9d09ff7ecfea515d5b415
Jul 29 22:46:36.393445 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:36.393423    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f404ba_6594_4814_b5c5_9904a205207c.slice/crio-9f171f2a2eaed45935c1f2948abba506ceb5306dc62beaa8632405ebca8823c8 WatchSource:0}: Error finding container 9f171f2a2eaed45935c1f2948abba506ceb5306dc62beaa8632405ebca8823c8: Status 404 returned error can't find the container with id 9f171f2a2eaed45935c1f2948abba506ceb5306dc62beaa8632405ebca8823c8
Jul 29 22:46:36.395787 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:46:36.395768    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8402f600_296c_4fa7_a6a2_1d34bb23781c.slice/crio-ee2e67146a88a9330a51e6e27127599278067dba105b0a06f290c79ee4381edd WatchSource:0}: Error finding container ee2e67146a88a9330a51e6e27127599278067dba105b0a06f290c79ee4381edd: Status 404 returned error can't find the container with id ee2e67146a88a9330a51e6e27127599278067dba105b0a06f290c79ee4381edd
Jul 29 22:46:36.468538 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.468511    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:36.468658 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:36.468627    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:46:36.478424 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.478383    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r4lqd" event={"ID":"f678c5dc-4755-4702-9be8-79557973646a","Type":"ContainerStarted","Data":"915d5875645f45b7e3cdc81b16c84ee8999a20e1420c041667e662dd960723ca"}
Jul 29 22:46:36.479701 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.479674    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fxc56" event={"ID":"6289a7d5-1978-462d-9ffc-a4135ff29c6d","Type":"ContainerStarted","Data":"03a78d026b57fc7b4e6e076f9b30cacc6279aed33db812b3263f3bc31d4dbe5a"}
Jul 29 22:46:36.480802 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.480777    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r8cns" event={"ID":"8402f600-296c-4fa7-a6a2-1d34bb23781c","Type":"ContainerStarted","Data":"ee2e67146a88a9330a51e6e27127599278067dba105b0a06f290c79ee4381edd"}
Jul 29 22:46:36.482059 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.482038    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95" event={"ID":"9e4e9b57-17f8-495a-b460-319701b58ba3","Type":"ContainerStarted","Data":"99a8250bea8ca81024e3081b8531ee0cf6cca01e1abf531af6543875e1f3a45c"}
Jul 29 22:46:36.483153 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.483123    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-86ljg" event={"ID":"442fac65-cba1-44e4-bd5b-33fc746a4e54","Type":"ContainerStarted","Data":"84b74b55d0c4ebdd506ddd98d99edde976312a466bedc6374f2b44a565da6192"}
Jul 29 22:46:36.484186 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.484164    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr" event={"ID":"9d48fa5a-4721-40b8-983e-c6febef5ea3b","Type":"ContainerStarted","Data":"68375381af9045ed4241105bc95cb4fc224d1cadd89b73538147025638c7728d"}
Jul 29 22:46:36.489004 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.488977    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal" event={"ID":"c3fee7606f52b4c69acd2656db749d26","Type":"ContainerStarted","Data":"a1bb301a78e48ea601497c8c772816933c94c437ed4fc1ac5afd55097213b625"}
Jul 29 22:46:36.490122 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.490100    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbdtd" event={"ID":"06f404ba-6594-4814-b5c5-9904a205207c","Type":"ContainerStarted","Data":"9f171f2a2eaed45935c1f2948abba506ceb5306dc62beaa8632405ebca8823c8"}
Jul 29 22:46:36.491116 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.491093    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shgn2" event={"ID":"5ab88d84-44ee-463c-986e-43d1da51284b","Type":"ContainerStarted","Data":"b12a315d4a0d163595dc8146cb75857a616cc2cb3dc9d09ff7ecfea515d5b415"}
Jul 29 22:46:36.492218 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.492197    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gjsxw" event={"ID":"8e491a98-c409-49a0-9510-8ed3d321366d","Type":"ContainerStarted","Data":"270383b29e6fa3c09d7635f676b77e109c69b115c4c87cf291966eb19dc71bba"}
Jul 29 22:46:36.501357 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:36.501300    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal" podStartSLOduration=2.501283687 podStartE2EDuration="2.501283687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:36.500681981 +0000 UTC m=+3.505879404" watchObservedRunningTime="2026-07-29 22:46:36.501283687 +0000 UTC m=+3.506481109"
Jul 29 22:46:37.023085 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:37.022711    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:37.023085 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:37.022937    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:37.023285 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:37.023220    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs podName:0d19dc00-e3da-4306-a5ff-8cca92e02a1c nodeName:}" failed. No retries permitted until 2026-07-29 22:46:39.023091623 +0000 UTC m=+6.028289040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs") pod "network-metrics-daemon-t5x5r" (UID: "0d19dc00-e3da-4306-a5ff-8cca92e02a1c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:37.023348 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:37.023296    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:37.023441 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:37.023425    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:37.023513 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:37.023472    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret podName:450449c0-a316-47ca-81cd-844420f36bea nodeName:}" failed. No retries permitted until 2026-07-29 22:46:39.023459006 +0000 UTC m=+6.028656412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret") pod "global-pull-secret-syncer-w9w6m" (UID: "450449c0-a316-47ca-81cd-844420f36bea") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:37.124640 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:37.124601    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlx5m\" (UniqueName: \"kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m\") pod \"network-check-target-54qd4\" (UID: \"2fcc94dd-27b8-4ee6-930c-be6245e627c0\") " pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:37.124837 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:37.124819    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:37.124908 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:37.124845    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:37.124908 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:37.124858    2569 projected.go:196] Error preparing data for projected volume kube-api-access-rlx5m for pod openshift-network-diagnostics/network-check-target-54qd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:37.125014 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:37.124916    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m podName:2fcc94dd-27b8-4ee6-930c-be6245e627c0 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:39.124897222 +0000 UTC m=+6.130094629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlx5m" (UniqueName: "kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m") pod "network-check-target-54qd4" (UID: "2fcc94dd-27b8-4ee6-930c-be6245e627c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:37.341625 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:37.341467    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 22:41:35 +0000 UTC" deadline="2028-01-04 20:57:07.608977489 +0000 UTC"
Jul 29 22:46:37.341625 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:37.341542    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12574h10m30.26743976s"
Jul 29 22:46:37.468848 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:37.468821    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:37.469020 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:37.468942    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:46:37.469020 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:37.469009    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:37.469141 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:37.469119    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:46:37.514386 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:37.514348    2569 generic.go:356] "Generic (PLEG): container finished" podID="21d832289f76f2c2f5062adee59e8cd8" containerID="386b606e813830bd7a56e9a0d79ecf1840596847e39a2976ff43905d5bf31a97" exitCode=0
Jul 29 22:46:37.515352 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:37.515327    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal" event={"ID":"21d832289f76f2c2f5062adee59e8cd8","Type":"ContainerDied","Data":"386b606e813830bd7a56e9a0d79ecf1840596847e39a2976ff43905d5bf31a97"}
Jul 29 22:46:38.469108 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:38.469072    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:38.469901 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:38.469226    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:46:38.534970 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:38.534934    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal" event={"ID":"21d832289f76f2c2f5062adee59e8cd8","Type":"ContainerStarted","Data":"94de78baceb3d2169a7f778752140b0ba9abf1051e85f767b6a617ea7f65e96b"}
Jul 29 22:46:38.551092 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:38.550268    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal" podStartSLOduration=4.550249067 podStartE2EDuration="4.550249067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:38.549943942 +0000 UTC m=+5.555141366" watchObservedRunningTime="2026-07-29 22:46:38.550249067 +0000 UTC m=+5.555446491"
Jul 29 22:46:39.042253 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:39.042213    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:39.042448 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:39.042276    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:39.042448 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:39.042402    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:39.042589 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:39.042458    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs podName:0d19dc00-e3da-4306-a5ff-8cca92e02a1c nodeName:}" failed. No retries permitted until 2026-07-29 22:46:43.042439915 +0000 UTC m=+10.047637330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs") pod "network-metrics-daemon-t5x5r" (UID: "0d19dc00-e3da-4306-a5ff-8cca92e02a1c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:39.042879 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:39.042857    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:39.042988 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:39.042913    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret podName:450449c0-a316-47ca-81cd-844420f36bea nodeName:}" failed. No retries permitted until 2026-07-29 22:46:43.042898893 +0000 UTC m=+10.048096295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret") pod "global-pull-secret-syncer-w9w6m" (UID: "450449c0-a316-47ca-81cd-844420f36bea") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:39.142763 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:39.142730    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlx5m\" (UniqueName: \"kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m\") pod \"network-check-target-54qd4\" (UID: \"2fcc94dd-27b8-4ee6-930c-be6245e627c0\") " pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:39.142949 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:39.142924    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:39.142949 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:39.142943    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:39.143069 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:39.142956    2569 projected.go:196] Error preparing data for projected volume kube-api-access-rlx5m for pod openshift-network-diagnostics/network-check-target-54qd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:39.143069 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:39.143025    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m podName:2fcc94dd-27b8-4ee6-930c-be6245e627c0 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:43.142997999 +0000 UTC m=+10.148195418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlx5m" (UniqueName: "kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m") pod "network-check-target-54qd4" (UID: "2fcc94dd-27b8-4ee6-930c-be6245e627c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:39.472529 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:39.471474    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:39.472529 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:39.471615    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:46:39.472529 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:39.471997    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:39.472529 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:39.472089    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:46:40.469084 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:40.469044    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:40.469260 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:40.469197    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:46:41.469228 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:41.469185    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:41.469663 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:41.469314    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:46:41.469861 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:41.469840    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:41.469965 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:41.469941    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:46:42.469365 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:42.469175    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:42.469837 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:42.469389    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:46:43.076743 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:43.076705    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:43.076930 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:43.076775    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:43.076930 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:43.076900    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:43.077053 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:43.076957    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs podName:0d19dc00-e3da-4306-a5ff-8cca92e02a1c nodeName:}" failed. No retries permitted until 2026-07-29 22:46:51.076939544 +0000 UTC m=+18.082136944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs") pod "network-metrics-daemon-t5x5r" (UID: "0d19dc00-e3da-4306-a5ff-8cca92e02a1c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:43.077053 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:43.077023    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:43.077053 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:43.077051    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret podName:450449c0-a316-47ca-81cd-844420f36bea nodeName:}" failed. No retries permitted until 2026-07-29 22:46:51.077043878 +0000 UTC m=+18.082241280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret") pod "global-pull-secret-syncer-w9w6m" (UID: "450449c0-a316-47ca-81cd-844420f36bea") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:43.177223 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:43.177185    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlx5m\" (UniqueName: \"kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m\") pod \"network-check-target-54qd4\" (UID: \"2fcc94dd-27b8-4ee6-930c-be6245e627c0\") " pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:43.177393 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:43.177332    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:43.177393 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:43.177361    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:43.177393 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:43.177375    2569 projected.go:196] Error preparing data for projected volume kube-api-access-rlx5m for pod openshift-network-diagnostics/network-check-target-54qd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:43.177587 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:43.177440    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m podName:2fcc94dd-27b8-4ee6-930c-be6245e627c0 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:51.177420506 +0000 UTC m=+18.182617927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlx5m" (UniqueName: "kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m") pod "network-check-target-54qd4" (UID: "2fcc94dd-27b8-4ee6-930c-be6245e627c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:43.470549 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:43.470191    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:43.470549 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:43.470321    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:46:43.471260 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:43.471096    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:43.471260 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:43.471214    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:46:44.470059 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:44.469518    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:44.470059 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:44.469667    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:46:45.469427 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:45.469397    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:45.469895 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:45.469534    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:46:45.469895 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:45.469722    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:45.469895 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:45.469830    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:46:46.468687 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:46.468653    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:46.468885 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:46.468784    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:46:47.469557 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:47.469511    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:47.470005 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:47.469523    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:47.470005 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:47.469650    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:46:47.470005 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:47.469711    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:46:48.469077 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:48.469041    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:48.469265 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:48.469160    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:46:49.468893 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:49.468855    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:49.469350 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:49.468992    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:46:49.469350 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:49.469045    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:49.469350 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:49.469150    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:46:50.468912 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:50.468876    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:50.469355 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:50.468974    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:46:51.138412 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:51.138372    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:51.138590 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:51.138453    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:51.138590 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:51.138563    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:51.138697 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:51.138594    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:51.138697 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:51.138639    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs podName:0d19dc00-e3da-4306-a5ff-8cca92e02a1c nodeName:}" failed. No retries permitted until 2026-07-29 22:47:07.138621257 +0000 UTC m=+34.143818662 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs") pod "network-metrics-daemon-t5x5r" (UID: "0d19dc00-e3da-4306-a5ff-8cca92e02a1c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:51.138697 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:51.138660    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret podName:450449c0-a316-47ca-81cd-844420f36bea nodeName:}" failed. No retries permitted until 2026-07-29 22:47:07.138649875 +0000 UTC m=+34.143847278 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret") pod "global-pull-secret-syncer-w9w6m" (UID: "450449c0-a316-47ca-81cd-844420f36bea") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:51.238804 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:51.238759    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlx5m\" (UniqueName: \"kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m\") pod \"network-check-target-54qd4\" (UID: \"2fcc94dd-27b8-4ee6-930c-be6245e627c0\") " pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:51.238985 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:51.238962    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:51.239074 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:51.238994    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:51.239074 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:51.239007    2569 projected.go:196] Error preparing data for projected volume kube-api-access-rlx5m for pod openshift-network-diagnostics/network-check-target-54qd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:51.239179 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:51.239087    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m podName:2fcc94dd-27b8-4ee6-930c-be6245e627c0 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:07.239065739 +0000 UTC m=+34.244263153 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlx5m" (UniqueName: "kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m") pod "network-check-target-54qd4" (UID: "2fcc94dd-27b8-4ee6-930c-be6245e627c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:51.469475 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:51.469444    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:51.469939 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:51.469601    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:51.469939 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:51.469595    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:46:51.469939 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:51.469719    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:46:52.468921 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:52.468883    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:52.469093 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:52.469039    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:46:53.470042 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:53.470004    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:53.470042 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:53.470042    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:53.470537 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:53.470126    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:46:53.470537 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:53.470257    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:46:54.469567 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:54.469534    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:54.469733 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:54.469671    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:46:55.469196 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.468813    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:55.469906 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.468834    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:55.469906 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:55.469288    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:46:55.469906 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:55.469440    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:46:55.570500 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.570452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r4lqd" event={"ID":"f678c5dc-4755-4702-9be8-79557973646a","Type":"ContainerStarted","Data":"cc797a5d70780b8a111ea0c4f98b1989e3f7ae12b645ff1037fae4f06195d313"}
Jul 29 22:46:55.573004 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.572972    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fxc56" event={"ID":"6289a7d5-1978-462d-9ffc-a4135ff29c6d","Type":"ContainerStarted","Data":"163951553bd661c51fe5a13d11d3bbec5eb599f141b920fd77f32ae821b9015e"}
Jul 29 22:46:55.575265 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.575207    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r8cns" event={"ID":"8402f600-296c-4fa7-a6a2-1d34bb23781c","Type":"ContainerStarted","Data":"81275c6e2e7336996c9d9a85ee5d162d61afdb728ce7cfae032fa0b27ff3c285"}
Jul 29 22:46:55.579001 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.578978    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95" event={"ID":"9e4e9b57-17f8-495a-b460-319701b58ba3","Type":"ContainerStarted","Data":"9675fc4a2fae4fed2c74ed4643ba87f82c08e8ac0c0609a348f9bdb0e79f76ee"}
Jul 29 22:46:55.579118 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.579062    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95" event={"ID":"9e4e9b57-17f8-495a-b460-319701b58ba3","Type":"ContainerStarted","Data":"6d4206329c0ab79d779de729aea19d2a8ab39b5ab64e95e6a285034561cace7c"}
Jul 29 22:46:55.579180 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.579116    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95" event={"ID":"9e4e9b57-17f8-495a-b460-319701b58ba3","Type":"ContainerStarted","Data":"17f202ca4dfa19eea51a90299c1f10e87b5d8add028d0538631d0de953d042ef"}
Jul 29 22:46:55.579180 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.579165    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95" event={"ID":"9e4e9b57-17f8-495a-b460-319701b58ba3","Type":"ContainerStarted","Data":"df258fbc84f635468066832222c2d83e8c2fdae4a2447def82e69907c4d8b6bf"}
Jul 29 22:46:55.579283 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.579211    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95" event={"ID":"9e4e9b57-17f8-495a-b460-319701b58ba3","Type":"ContainerStarted","Data":"3e54df1f2dc5ed8229851606ef0166b863d88ceeedf4af4f63f929b00dd22a85"}
Jul 29 22:46:55.579283 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.579261    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95" event={"ID":"9e4e9b57-17f8-495a-b460-319701b58ba3","Type":"ContainerStarted","Data":"f95f7d38e425c467f98cb684cb59319982126a544e57130f3d9161fdbdf41105"}
Jul 29 22:46:55.580956 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.580908    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-86ljg" event={"ID":"442fac65-cba1-44e4-bd5b-33fc746a4e54","Type":"ContainerStarted","Data":"c8e51582d84b638f70b2e73503c57be500fc9f4b9940c301d0e28b4be4258116"}
Jul 29 22:46:55.582542 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.582517    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr" event={"ID":"9d48fa5a-4721-40b8-983e-c6febef5ea3b","Type":"ContainerStarted","Data":"6893f4a6750deea3f0b11648ada53685f8cae92beffa8500d89beb661efefdfe"}
Jul 29 22:46:55.584032 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.584004    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbdtd" event={"ID":"06f404ba-6594-4814-b5c5-9904a205207c","Type":"ContainerStarted","Data":"62bb030b50fbe77752c7e7a45a3255e90897f28a29d75001d06491676edc2d08"}
Jul 29 22:46:55.585701 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.585674    2569 generic.go:356] "Generic (PLEG): container finished" podID="5ab88d84-44ee-463c-986e-43d1da51284b" containerID="34201cb5da9fcfe7b89771c6568fb0db012687c3a4650a7ca582e65ed0b4eb67" exitCode=0
Jul 29 22:46:55.585825 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.585721    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shgn2" event={"ID":"5ab88d84-44ee-463c-986e-43d1da51284b","Type":"ContainerDied","Data":"34201cb5da9fcfe7b89771c6568fb0db012687c3a4650a7ca582e65ed0b4eb67"}
Jul 29 22:46:55.587505 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.587456    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r4lqd" podStartSLOduration=8.504748929 podStartE2EDuration="22.587445591s" totalImagesPullingTime="14.082696662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:55.586875533 +0000 UTC m=+22.592072970" watchObservedRunningTime="2026-07-29 22:46:55.587445591 +0000 UTC m=+22.592643014"
Jul 29 22:46:55.646440 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.646396    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-86ljg" podStartSLOduration=4.233718964 podStartE2EDuration="22.64638127s" totalImagesPullingTime="18.412662306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:55.632226262 +0000 UTC m=+22.637423686" watchObservedRunningTime="2026-07-29 22:46:55.64638127 +0000 UTC m=+22.651578693"
Jul 29 22:46:55.660668 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.660629    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r8cns" podStartSLOduration=4.257641168 podStartE2EDuration="22.660615351s" totalImagesPullingTime="18.402974183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:55.646506813 +0000 UTC m=+22.651704232" watchObservedRunningTime="2026-07-29 22:46:55.660615351 +0000 UTC m=+22.665812774"
Jul 29 22:46:55.660821 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.660802    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mbdtd" podStartSLOduration=4.253966911 podStartE2EDuration="22.660798464s" totalImagesPullingTime="18.406831553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:55.660609921 +0000 UTC m=+22.665807343" watchObservedRunningTime="2026-07-29 22:46:55.660798464 +0000 UTC m=+22.665995887"
Jul 29 22:46:55.680001 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:55.679963    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fxc56" podStartSLOduration=4.202862062 podStartE2EDuration="22.679950405s" totalImagesPullingTime="18.477088343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:55.679791721 +0000 UTC m=+22.684989141" watchObservedRunningTime="2026-07-29 22:46:55.679950405 +0000 UTC m=+22.685147828"
Jul 29 22:46:56.172985 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:56.172959    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 22:46:56.175064 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:56.175040    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r4lqd"
Jul 29 22:46:56.175639 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:56.175622    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r4lqd"
Jul 29 22:46:56.372395 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:56.372176    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T22:46:56.172981762Z","UUID":"16c2aafa-7192-4e2d-867a-831575ee504f","Handler":null,"Name":"","Endpoint":""}
Jul 29 22:46:56.373938 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:56.373920    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 22:46:56.374074 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:56.373961    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 22:46:56.469551 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:56.469517    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:56.470112 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:56.469630    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:46:56.590702 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:56.590663    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr" event={"ID":"9d48fa5a-4721-40b8-983e-c6febef5ea3b","Type":"ContainerStarted","Data":"ccee26af02dbeb85ca2050f39f287c2f29e7f12aee5924503f1b3092d20958bb"}
Jul 29 22:46:56.592427 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:56.592364    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gjsxw" event={"ID":"8e491a98-c409-49a0-9510-8ed3d321366d","Type":"ContainerStarted","Data":"05635587a8d5a8d84d0925baf31002fe9550011dbf9cf81178f848a2728e1bac"}
Jul 29 22:46:56.606595 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:56.606549    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gjsxw" podStartSLOduration=5.1916743069999995 podStartE2EDuration="23.606532801s" totalImagesPullingTime="18.414858494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:56.606341749 +0000 UTC m=+23.611539172" watchObservedRunningTime="2026-07-29 22:46:56.606532801 +0000 UTC m=+23.611730223"
Jul 29 22:46:57.469525 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:57.469174    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:57.469525 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:57.469318    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:57.469525 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:57.469420    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:46:57.469525 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:57.469311    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:46:57.597331 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:57.597302    2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 29 22:46:57.597331 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:57.597320    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr" event={"ID":"9d48fa5a-4721-40b8-983e-c6febef5ea3b","Type":"ContainerStarted","Data":"56bdfbfcb48d46dd63b083e5baf4f5143754c3c0e17b548ea20711d8e0069ddc"}
Jul 29 22:46:57.615638 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:57.615525    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47wpr" podStartSLOduration=3.942624614 podStartE2EDuration="24.615508377s" totalImagesPullingTime="20.672883763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:57.615079023 +0000 UTC m=+24.620276445" watchObservedRunningTime="2026-07-29 22:46:57.615508377 +0000 UTC m=+24.620705800"
Jul 29 22:46:58.468885 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:58.468854    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:46:58.469042 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:58.468994    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:46:58.606680 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:58.606636    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95" event={"ID":"9e4e9b57-17f8-495a-b460-319701b58ba3","Type":"ContainerStarted","Data":"6526888ffaaeb37082668e790c25366fe8065ee6c9513aba1e493f348b6f7672"}
Jul 29 22:46:59.469323 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:59.469286    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:46:59.469503 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:46:59.469343    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:46:59.469503 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:59.469432    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:46:59.469613 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:46:59.469559    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:47:00.469669 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:00.469494    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:47:00.470245 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:00.469748    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:47:00.612289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:00.612254    2569 generic.go:356] "Generic (PLEG): container finished" podID="5ab88d84-44ee-463c-986e-43d1da51284b" containerID="3e469bf9fe209e584df82ca7b5394ba3ea4af638c0c987251d313b60c2f1b2c3" exitCode=0
Jul 29 22:47:00.612446 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:00.612339    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shgn2" event={"ID":"5ab88d84-44ee-463c-986e-43d1da51284b","Type":"ContainerDied","Data":"3e469bf9fe209e584df82ca7b5394ba3ea4af638c0c987251d313b60c2f1b2c3"}
Jul 29 22:47:00.615965 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:00.615945    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95" event={"ID":"9e4e9b57-17f8-495a-b460-319701b58ba3","Type":"ContainerStarted","Data":"aa5c2743a06b4f2317ed3993725d47cc51d778ff411f23b490a5841ebbd442e8"}
Jul 29 22:47:00.616235 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:00.616218    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:47:00.616308 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:00.616276    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:47:00.631629 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:00.631609    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:47:00.655435 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:00.655396    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95" podStartSLOduration=9.169330698 podStartE2EDuration="27.655381908s" totalImagesPullingTime="18.48605121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:47:00.654105496 +0000 UTC m=+27.659302918" watchObservedRunningTime="2026-07-29 22:47:00.655381908 +0000 UTC m=+27.660579331"
Jul 29 22:47:00.789602 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:00.789520    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r4lqd"
Jul 29 22:47:00.789734 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:00.789649    2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 29 22:47:00.790396 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:00.790377    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r4lqd"
Jul 29 22:47:01.468783 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:01.468652    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:47:01.468949 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:01.468689    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:47:01.468949 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:01.468861    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:47:01.469068 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:01.468966    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:47:01.619634 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:01.619582    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:47:01.636806 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:01.636779    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:47:01.896692 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:01.896661    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-54qd4"]
Jul 29 22:47:01.896886 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:01.896790    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:47:01.897001 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:01.896889    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:47:01.901566 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:01.901540    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w9w6m"]
Jul 29 22:47:01.901689 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:01.901636    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:47:01.901754 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:01.901717    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:47:01.908249 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:01.908223    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t5x5r"]
Jul 29 22:47:01.908360 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:01.908349    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:47:01.908494 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:01.908463    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:47:02.625109 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:02.625077    2569 generic.go:356] "Generic (PLEG): container finished" podID="5ab88d84-44ee-463c-986e-43d1da51284b" containerID="cc32b8f063020928fcf2921f1da0da150735b679c2065e0dd965909a1871cb78" exitCode=0
Jul 29 22:47:02.625570 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:02.625163    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shgn2" event={"ID":"5ab88d84-44ee-463c-986e-43d1da51284b","Type":"ContainerDied","Data":"cc32b8f063020928fcf2921f1da0da150735b679c2065e0dd965909a1871cb78"}
Jul 29 22:47:03.469771 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:03.469738    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:47:03.469940 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:03.469775    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:47:03.469940 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:03.469853    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:47:03.470076 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:03.469924    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:47:03.470076 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:03.469946    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:47:03.470076 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:03.470023    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:47:04.638540 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:04.638313    2569 generic.go:356] "Generic (PLEG): container finished" podID="5ab88d84-44ee-463c-986e-43d1da51284b" containerID="85c79e8b125297eb39e35924bf15727bb0b6f8326436646ba37033023091c1bd" exitCode=0
Jul 29 22:47:04.638989 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:04.638384    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shgn2" event={"ID":"5ab88d84-44ee-463c-986e-43d1da51284b","Type":"ContainerDied","Data":"85c79e8b125297eb39e35924bf15727bb0b6f8326436646ba37033023091c1bd"}
Jul 29 22:47:05.468868 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:05.468827    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:47:05.468868 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:05.468860    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:47:05.468868 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:05.468837    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:47:05.469205 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:05.468978    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:47:05.469205 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:05.469061    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:47:05.469205 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:05.469098    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:47:07.150836 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:07.150796    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:47:07.151353 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:07.150865    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:47:07.151353 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:07.150976    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:47:07.151353 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:07.150979    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:47:07.151353 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:07.151038    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret podName:450449c0-a316-47ca-81cd-844420f36bea nodeName:}" failed. No retries permitted until 2026-07-29 22:47:39.151021188 +0000 UTC m=+66.156218589 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret") pod "global-pull-secret-syncer-w9w6m" (UID: "450449c0-a316-47ca-81cd-844420f36bea") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:47:07.151353 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:07.151052    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs podName:0d19dc00-e3da-4306-a5ff-8cca92e02a1c nodeName:}" failed. No retries permitted until 2026-07-29 22:47:39.151045611 +0000 UTC m=+66.156243012 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs") pod "network-metrics-daemon-t5x5r" (UID: "0d19dc00-e3da-4306-a5ff-8cca92e02a1c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:47:07.252206 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:07.252170    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlx5m\" (UniqueName: \"kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m\") pod \"network-check-target-54qd4\" (UID: \"2fcc94dd-27b8-4ee6-930c-be6245e627c0\") " pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:47:07.252364 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:07.252330    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:47:07.252364 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:07.252351    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:47:07.252364 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:07.252365    2569 projected.go:196] Error preparing data for projected volume kube-api-access-rlx5m for pod openshift-network-diagnostics/network-check-target-54qd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:47:07.252536 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:07.252429    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m podName:2fcc94dd-27b8-4ee6-930c-be6245e627c0 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:39.252410955 +0000 UTC m=+66.257608362 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlx5m" (UniqueName: "kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m") pod "network-check-target-54qd4" (UID: "2fcc94dd-27b8-4ee6-930c-be6245e627c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:47:07.468877 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:07.468840    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:47:07.469034 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:07.468980    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9w6m" podUID="450449c0-a316-47ca-81cd-844420f36bea"
Jul 29 22:47:07.469115 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:07.469075    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:47:07.469198 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:07.469173    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:47:07.469305 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:07.469225    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:47:07.469305 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:07.469289    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-54qd4" podUID="2fcc94dd-27b8-4ee6-930c-be6245e627c0"
Jul 29 22:47:08.649461 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.649431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shgn2" event={"ID":"5ab88d84-44ee-463c-986e-43d1da51284b","Type":"ContainerStarted","Data":"0c30472d6872e72043b23ec8692cfe7e0b2a967f8c8ffcea0c81645ba9704a68"}
Jul 29 22:47:08.850818 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.850736    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeReady"
Jul 29 22:47:08.850947 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.850866    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 22:47:08.897540 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.897504    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xxsch"]
Jul 29 22:47:08.900122 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.900095    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:08.900234 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.900163    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wf4k2"]
Jul 29 22:47:08.902025 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.902004    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jtttq\""
Jul 29 22:47:08.902120 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.902005    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 22:47:08.902427 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.902412    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 22:47:08.903184 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.903168    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wf4k2"
Jul 29 22:47:08.905172 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.905156    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 22:47:08.905720 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.905705    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 22:47:08.905781 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.905742    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 22:47:08.905821 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.905807    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d8h7n\""
Jul 29 22:47:08.909945 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.909925    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xxsch"]
Jul 29 22:47:08.912973 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:08.912950    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wf4k2"]
Jul 29 22:47:09.067115 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.067078    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtljz\" (UniqueName: \"kubernetes.io/projected/49ebc216-8a22-460c-b045-06da7890eba1-kube-api-access-dtljz\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:09.067115 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.067128    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49ebc216-8a22-460c-b045-06da7890eba1-config-volume\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:09.067357 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.067145    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ebc216-8a22-460c-b045-06da7890eba1-tmp-dir\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:09.067357 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.067180    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sf7d\" (UniqueName: \"kubernetes.io/projected/bc372877-f198-4771-b46c-8ebd74cca4bc-kube-api-access-4sf7d\") pod \"ingress-canary-wf4k2\" (UID: \"bc372877-f198-4771-b46c-8ebd74cca4bc\") " pod="openshift-ingress-canary/ingress-canary-wf4k2"
Jul 29 22:47:09.067357 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.067261    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:09.067357 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.067297    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc372877-f198-4771-b46c-8ebd74cca4bc-cert\") pod \"ingress-canary-wf4k2\" (UID: \"bc372877-f198-4771-b46c-8ebd74cca4bc\") " pod="openshift-ingress-canary/ingress-canary-wf4k2"
Jul 29 22:47:09.167901 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.167820    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:09.167901 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.167858    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc372877-f198-4771-b46c-8ebd74cca4bc-cert\") pod \"ingress-canary-wf4k2\" (UID: \"bc372877-f198-4771-b46c-8ebd74cca4bc\") " pod="openshift-ingress-canary/ingress-canary-wf4k2"
Jul 29 22:47:09.168055 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.167915    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtljz\" (UniqueName: \"kubernetes.io/projected/49ebc216-8a22-460c-b045-06da7890eba1-kube-api-access-dtljz\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:09.168055 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.167944    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49ebc216-8a22-460c-b045-06da7890eba1-config-volume\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:09.168055 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.167959    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ebc216-8a22-460c-b045-06da7890eba1-tmp-dir\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:09.168055 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:09.167962    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:47:09.168055 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.167977    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sf7d\" (UniqueName: \"kubernetes.io/projected/bc372877-f198-4771-b46c-8ebd74cca4bc-kube-api-access-4sf7d\") pod \"ingress-canary-wf4k2\" (UID: \"bc372877-f198-4771-b46c-8ebd74cca4bc\") " pod="openshift-ingress-canary/ingress-canary-wf4k2"
Jul 29 22:47:09.168055 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:09.168030    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls podName:49ebc216-8a22-460c-b045-06da7890eba1 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:09.668009784 +0000 UTC m=+36.673207185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls") pod "dns-default-xxsch" (UID: "49ebc216-8a22-460c-b045-06da7890eba1") : secret "dns-default-metrics-tls" not found
Jul 29 22:47:09.168376 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.168354    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ebc216-8a22-460c-b045-06da7890eba1-tmp-dir\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:09.168524 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.168506    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49ebc216-8a22-460c-b045-06da7890eba1-config-volume\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:09.171996 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.171978    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc372877-f198-4771-b46c-8ebd74cca4bc-cert\") pod \"ingress-canary-wf4k2\" (UID: \"bc372877-f198-4771-b46c-8ebd74cca4bc\") " pod="openshift-ingress-canary/ingress-canary-wf4k2"
Jul 29 22:47:09.178130 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.178096    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtljz\" (UniqueName: \"kubernetes.io/projected/49ebc216-8a22-460c-b045-06da7890eba1-kube-api-access-dtljz\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:09.178279 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.178257    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sf7d\" (UniqueName: \"kubernetes.io/projected/bc372877-f198-4771-b46c-8ebd74cca4bc-kube-api-access-4sf7d\") pod \"ingress-canary-wf4k2\" (UID: \"bc372877-f198-4771-b46c-8ebd74cca4bc\") " pod="openshift-ingress-canary/ingress-canary-wf4k2"
Jul 29 22:47:09.216575 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.216546    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wf4k2"
Jul 29 22:47:09.347737 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.347549    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wf4k2"]
Jul 29 22:47:09.351336 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:47:09.351304    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc372877_f198_4771_b46c_8ebd74cca4bc.slice/crio-45cabfb3ee63786fe89efa47af6b591120e8d82d39093a66e2c2d80356fcee58 WatchSource:0}: Error finding container 45cabfb3ee63786fe89efa47af6b591120e8d82d39093a66e2c2d80356fcee58: Status 404 returned error can't find the container with id 45cabfb3ee63786fe89efa47af6b591120e8d82d39093a66e2c2d80356fcee58
Jul 29 22:47:09.468969 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.468933    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:47:09.469137 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.468938    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:47:09.469137 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.469052    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:47:09.471609 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.471579    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 22:47:09.471738 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.471625    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 22:47:09.471883 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.471858    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-csbtv\""
Jul 29 22:47:09.471883 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.471874    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 22:47:09.472061 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.471859    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wkbjs\""
Jul 29 22:47:09.472061 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.471960    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 22:47:09.652873 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.652827    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wf4k2" event={"ID":"bc372877-f198-4771-b46c-8ebd74cca4bc","Type":"ContainerStarted","Data":"45cabfb3ee63786fe89efa47af6b591120e8d82d39093a66e2c2d80356fcee58"}
Jul 29 22:47:09.655290 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.655264    2569 generic.go:356] "Generic (PLEG): container finished" podID="5ab88d84-44ee-463c-986e-43d1da51284b" containerID="0c30472d6872e72043b23ec8692cfe7e0b2a967f8c8ffcea0c81645ba9704a68" exitCode=0
Jul 29 22:47:09.655411 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.655324    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shgn2" event={"ID":"5ab88d84-44ee-463c-986e-43d1da51284b","Type":"ContainerDied","Data":"0c30472d6872e72043b23ec8692cfe7e0b2a967f8c8ffcea0c81645ba9704a68"}
Jul 29 22:47:09.672441 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:09.672417    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:09.672555 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:09.672541    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:47:09.672610 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:09.672597    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls podName:49ebc216-8a22-460c-b045-06da7890eba1 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:10.672578615 +0000 UTC m=+37.677776018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls") pod "dns-default-xxsch" (UID: "49ebc216-8a22-460c-b045-06da7890eba1") : secret "dns-default-metrics-tls" not found
Jul 29 22:47:10.661784 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:10.661747    2569 generic.go:356] "Generic (PLEG): container finished" podID="5ab88d84-44ee-463c-986e-43d1da51284b" containerID="d25718acad7d56ae03d8e3cf90992dbe1dc054396a1ddd791b1a8c4e4a87f3ff" exitCode=0
Jul 29 22:47:10.662290 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:10.661797    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shgn2" event={"ID":"5ab88d84-44ee-463c-986e-43d1da51284b","Type":"ContainerDied","Data":"d25718acad7d56ae03d8e3cf90992dbe1dc054396a1ddd791b1a8c4e4a87f3ff"}
Jul 29 22:47:10.681715 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:10.681681    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:10.681901 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:10.681835    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:47:10.681969 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:10.681907    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls podName:49ebc216-8a22-460c-b045-06da7890eba1 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:12.68188279 +0000 UTC m=+39.687080199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls") pod "dns-default-xxsch" (UID: "49ebc216-8a22-460c-b045-06da7890eba1") : secret "dns-default-metrics-tls" not found
Jul 29 22:47:11.667570 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:11.667377    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shgn2" event={"ID":"5ab88d84-44ee-463c-986e-43d1da51284b","Type":"ContainerStarted","Data":"57b129f6e68399ac936892713a0d5d3c848005341ea70f36cd886f298e40005d"}
Jul 29 22:47:11.688131 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:11.688084    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-shgn2" podStartSLOduration=10.254745324 podStartE2EDuration="38.688069734s" totalImagesPullingTime="28.43332441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:47:11.687247106 +0000 UTC m=+38.692444529" watchObservedRunningTime="2026-07-29 22:47:11.688069734 +0000 UTC m=+38.693267156"
Jul 29 22:47:12.671137 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:12.671101    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wf4k2" event={"ID":"bc372877-f198-4771-b46c-8ebd74cca4bc","Type":"ContainerStarted","Data":"4f03d662175376bd88dba008df6562ed82dc3bd2268ac97d8c1f24f23f8bef0a"}
Jul 29 22:47:12.686848 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:12.686803    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wf4k2" podStartSLOduration=2.130452879 podStartE2EDuration="4.686787446s" totalImagesPullingTime="2.556334567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:47:12.686244452 +0000 UTC m=+39.691441875" watchObservedRunningTime="2026-07-29 22:47:12.686787446 +0000 UTC m=+39.691984868"
Jul 29 22:47:12.698559 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:12.698526    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:12.698675 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:12.698630    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:47:12.698737 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:12.698679    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls podName:49ebc216-8a22-460c-b045-06da7890eba1 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:16.698666282 +0000 UTC m=+43.703863683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls") pod "dns-default-xxsch" (UID: "49ebc216-8a22-460c-b045-06da7890eba1") : secret "dns-default-metrics-tls" not found
Jul 29 22:47:16.726888 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:16.726847    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:16.727241 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:16.726974    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:47:16.727241 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:16.727037    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls podName:49ebc216-8a22-460c-b045-06da7890eba1 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:24.727018118 +0000 UTC m=+51.732215522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls") pod "dns-default-xxsch" (UID: "49ebc216-8a22-460c-b045-06da7890eba1") : secret "dns-default-metrics-tls" not found
Jul 29 22:47:24.780583 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:24.780541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:24.781161 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:24.780657    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:47:24.781161 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:24.780710    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls podName:49ebc216-8a22-460c-b045-06da7890eba1 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:40.780697114 +0000 UTC m=+67.785894518 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls") pod "dns-default-xxsch" (UID: "49ebc216-8a22-460c-b045-06da7890eba1") : secret "dns-default-metrics-tls" not found
Jul 29 22:47:33.651032 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:33.651004    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvw95"
Jul 29 22:47:39.177956 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.177413    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:47:39.178546 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.178053    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:47:39.180613 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.180583    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 22:47:39.180741 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.180630    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 22:47:39.188307 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:39.188285    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 22:47:39.188399 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:39.188388    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs podName:0d19dc00-e3da-4306-a5ff-8cca92e02a1c nodeName:}" failed. No retries permitted until 2026-07-29 22:48:43.188367269 +0000 UTC m=+130.193564687 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs") pod "network-metrics-daemon-t5x5r" (UID: "0d19dc00-e3da-4306-a5ff-8cca92e02a1c") : secret "metrics-daemon-secret" not found
Jul 29 22:47:39.192081 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.192056    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/450449c0-a316-47ca-81cd-844420f36bea-original-pull-secret\") pod \"global-pull-secret-syncer-w9w6m\" (UID: \"450449c0-a316-47ca-81cd-844420f36bea\") " pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:47:39.278811 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.278774    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlx5m\" (UniqueName: \"kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m\") pod \"network-check-target-54qd4\" (UID: \"2fcc94dd-27b8-4ee6-930c-be6245e627c0\") " pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:47:39.281224 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.281203    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 22:47:39.291242 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.291218    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 22:47:39.302988 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.302958    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlx5m\" (UniqueName: \"kubernetes.io/projected/2fcc94dd-27b8-4ee6-930c-be6245e627c0-kube-api-access-rlx5m\") pod \"network-check-target-54qd4\" (UID: \"2fcc94dd-27b8-4ee6-930c-be6245e627c0\") " pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:47:39.481575 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.481546    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-csbtv\""
Jul 29 22:47:39.489546 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.489528    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:47:39.489637 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.489618    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9w6m"
Jul 29 22:47:39.639569 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.639536    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w9w6m"]
Jul 29 22:47:39.642194 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:47:39.642164    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450449c0_a316_47ca_81cd_844420f36bea.slice/crio-42413984d13cc5008ed2d929a4b26a7d59c6eb6b1ddc197762777494adf30cbe WatchSource:0}: Error finding container 42413984d13cc5008ed2d929a4b26a7d59c6eb6b1ddc197762777494adf30cbe: Status 404 returned error can't find the container with id 42413984d13cc5008ed2d929a4b26a7d59c6eb6b1ddc197762777494adf30cbe
Jul 29 22:47:39.644812 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.644788    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-54qd4"]
Jul 29 22:47:39.647824 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:47:39.647800    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcc94dd_27b8_4ee6_930c_be6245e627c0.slice/crio-7ea38d81dc874de00e668ac717e83e8b233e4d15037b605dca487f0029d386ad WatchSource:0}: Error finding container 7ea38d81dc874de00e668ac717e83e8b233e4d15037b605dca487f0029d386ad: Status 404 returned error can't find the container with id 7ea38d81dc874de00e668ac717e83e8b233e4d15037b605dca487f0029d386ad
Jul 29 22:47:39.734649 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.734554    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w9w6m" event={"ID":"450449c0-a316-47ca-81cd-844420f36bea","Type":"ContainerStarted","Data":"42413984d13cc5008ed2d929a4b26a7d59c6eb6b1ddc197762777494adf30cbe"}
Jul 29 22:47:39.735679 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:39.735654    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-54qd4" event={"ID":"2fcc94dd-27b8-4ee6-930c-be6245e627c0","Type":"ContainerStarted","Data":"7ea38d81dc874de00e668ac717e83e8b233e4d15037b605dca487f0029d386ad"}
Jul 29 22:47:40.790774 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:40.790732    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:47:40.791391 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:40.790896    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:47:40.791391 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:47:40.790963    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls podName:49ebc216-8a22-460c-b045-06da7890eba1 nodeName:}" failed. No retries permitted until 2026-07-29 22:48:12.790942987 +0000 UTC m=+99.796140387 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls") pod "dns-default-xxsch" (UID: "49ebc216-8a22-460c-b045-06da7890eba1") : secret "dns-default-metrics-tls" not found
Jul 29 22:47:44.750183 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:44.750141    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w9w6m" event={"ID":"450449c0-a316-47ca-81cd-844420f36bea","Type":"ContainerStarted","Data":"1194de27a19b1ee6c864118426ee703f4b38b565c33f76df0f4be9418c5d676a"}
Jul 29 22:47:44.751600 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:44.751573    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-54qd4" event={"ID":"2fcc94dd-27b8-4ee6-930c-be6245e627c0","Type":"ContainerStarted","Data":"c6ea700a0893983e7ea7b62d8de5d3f7ecc13ea07c95b44f45bbcfe3df2a7195"}
Jul 29 22:47:44.751721 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:44.751705    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:47:44.777230 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:44.777175    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w9w6m" podStartSLOduration=67.202252926 podStartE2EDuration="1m11.777160776s" totalImagesPullingTime="4.57490785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:47:44.763666904 +0000 UTC m=+71.768864328" watchObservedRunningTime="2026-07-29 22:47:44.777160776 +0000 UTC m=+71.782358200"
Jul 29 22:47:44.777455 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:44.777424    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-54qd4" podStartSLOduration=67.211048927 podStartE2EDuration="1m11.777414202s" totalImagesPullingTime="4.566365275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:47:44.77697083 +0000 UTC m=+71.782168253" watchObservedRunningTime="2026-07-29 22:47:44.777414202 +0000 UTC m=+71.782611626"
Jul 29 22:47:45.663881 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.663847    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"]
Jul 29 22:47:45.666913 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.666892    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.668872 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.668849    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 22:47:45.668981 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.668851    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 29 22:47:45.669131 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.669118    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 29 22:47:45.669177 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.669117    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 22:47:45.669460 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.669443    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 29 22:47:45.669627 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.669610    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 22:47:45.669950 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.669937    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 29 22:47:45.679782 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.679753    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"]
Jul 29 22:47:45.724561 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.724531    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.724713 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.724565    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.724713 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.724583    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flpdd\" (UniqueName: \"kubernetes.io/projected/968c59fb-1819-4280-8671-cd892f0d060a-kube-api-access-flpdd\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.724713 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.724660    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-ca\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.724713 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.724691    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-hub\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.724865 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.724713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/968c59fb-1819-4280-8671-cd892f0d060a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.825083 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.825048    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-hub\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.825601 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.825094    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/968c59fb-1819-4280-8671-cd892f0d060a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.825601 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.825238    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.825601 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.825283    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.825601 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.825305    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flpdd\" (UniqueName: \"kubernetes.io/projected/968c59fb-1819-4280-8671-cd892f0d060a-kube-api-access-flpdd\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.825601 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.825371    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-ca\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.825982 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.825957    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/968c59fb-1819-4280-8671-cd892f0d060a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.827714 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.827683    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-ca\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.827835 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.827743    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.827835 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.827772    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-hub\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.827942 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.827839    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.832055 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.832032    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flpdd\" (UniqueName: \"kubernetes.io/projected/968c59fb-1819-4280-8671-cd892f0d060a-kube-api-access-flpdd\") pod \"cluster-proxy-proxy-agent-5dd8d8f59c-hl447\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:45.991282 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:45.991248    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:47:46.120606 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:46.120569    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"]
Jul 29 22:47:46.125014 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:47:46.124977    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod968c59fb_1819_4280_8671_cd892f0d060a.slice/crio-cebac003a21e038609c9b2af0e5324354576bac7388e60bb7627c5b899902b3c WatchSource:0}: Error finding container cebac003a21e038609c9b2af0e5324354576bac7388e60bb7627c5b899902b3c: Status 404 returned error can't find the container with id cebac003a21e038609c9b2af0e5324354576bac7388e60bb7627c5b899902b3c
Jul 29 22:47:46.757037 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:46.757001    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" event={"ID":"968c59fb-1819-4280-8671-cd892f0d060a","Type":"ContainerStarted","Data":"cebac003a21e038609c9b2af0e5324354576bac7388e60bb7627c5b899902b3c"}
Jul 29 22:47:49.766199 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:49.766167    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" event={"ID":"968c59fb-1819-4280-8671-cd892f0d060a","Type":"ContainerStarted","Data":"7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55"}
Jul 29 22:47:50.772187 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:50.772151    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" event={"ID":"968c59fb-1819-4280-8671-cd892f0d060a","Type":"ContainerStarted","Data":"73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd"}
Jul 29 22:47:50.772187 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:50.772193    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" event={"ID":"968c59fb-1819-4280-8671-cd892f0d060a","Type":"ContainerStarted","Data":"dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04"}
Jul 29 22:47:50.790118 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:47:50.790069    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" podStartSLOduration=2.235562415 podStartE2EDuration="5.790055402s" totalImagesPullingTime="3.554492987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:47:50.789340344 +0000 UTC m=+77.794537766" watchObservedRunningTime="2026-07-29 22:47:50.790055402 +0000 UTC m=+77.795252824"
Jul 29 22:48:12.806082 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:48:12.806026    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:48:12.806563 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:48:12.806188    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:48:12.806563 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:48:12.806264    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls podName:49ebc216-8a22-460c-b045-06da7890eba1 nodeName:}" failed. No retries permitted until 2026-07-29 22:49:16.80624842 +0000 UTC m=+163.811445822 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls") pod "dns-default-xxsch" (UID: "49ebc216-8a22-460c-b045-06da7890eba1") : secret "dns-default-metrics-tls" not found
Jul 29 22:48:15.755774 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:48:15.755744    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-54qd4"
Jul 29 22:48:43.217253 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:48:43.217209    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:48:43.217762 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:48:43.217360    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 22:48:43.217762 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:48:43.217420    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs podName:0d19dc00-e3da-4306-a5ff-8cca92e02a1c nodeName:}" failed. No retries permitted until 2026-07-29 22:50:45.217405307 +0000 UTC m=+252.222602711 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs") pod "network-metrics-daemon-t5x5r" (UID: "0d19dc00-e3da-4306-a5ff-8cca92e02a1c") : secret "metrics-daemon-secret" not found
Jul 29 22:49:11.910031 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:49:11.909987    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xxsch" podUID="49ebc216-8a22-460c-b045-06da7890eba1"
Jul 29 22:49:11.990990 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:49:11.990960    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxsch"
Jul 29 22:49:12.485472 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:49:12.485420    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:49:16.850849 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:49:16.850800    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:49:16.851350 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:49:16.850945    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:49:16.851350 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:49:16.851008    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls podName:49ebc216-8a22-460c-b045-06da7890eba1 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:18.850993445 +0000 UTC m=+285.856190847 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls") pod "dns-default-xxsch" (UID: "49ebc216-8a22-460c-b045-06da7890eba1") : secret "dns-default-metrics-tls" not found
Jul 29 22:49:27.469156 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:49:27.469069    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:50:44.256916 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.256882    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-27xfm"]
Jul 29 22:50:44.259841 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.259823    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:44.262187 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.262161    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-6p84b\""
Jul 29 22:50:44.262769 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.262751    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 29 22:50:44.262867 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.262758    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 29 22:50:44.262867 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.262758    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 29 22:50:44.262867 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.262775    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 29 22:50:44.269530 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.269508    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 29 22:50:44.273750 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.273727    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-27xfm"]
Jul 29 22:50:44.326692 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.326657    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0f949f6-90cf-4e28-a713-3587649d077c-config\") pod \"console-operator-584dcb6b54-27xfm\" (UID: \"c0f949f6-90cf-4e28-a713-3587649d077c\") " pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:44.326692 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.326696    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0f949f6-90cf-4e28-a713-3587649d077c-trusted-ca\") pod \"console-operator-584dcb6b54-27xfm\" (UID: \"c0f949f6-90cf-4e28-a713-3587649d077c\") " pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:44.326950 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.326719    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kssjb\" (UniqueName: \"kubernetes.io/projected/c0f949f6-90cf-4e28-a713-3587649d077c-kube-api-access-kssjb\") pod \"console-operator-584dcb6b54-27xfm\" (UID: \"c0f949f6-90cf-4e28-a713-3587649d077c\") " pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:44.326950 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.326870    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0f949f6-90cf-4e28-a713-3587649d077c-serving-cert\") pod \"console-operator-584dcb6b54-27xfm\" (UID: \"c0f949f6-90cf-4e28-a713-3587649d077c\") " pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:44.427644 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.427612    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kssjb\" (UniqueName: \"kubernetes.io/projected/c0f949f6-90cf-4e28-a713-3587649d077c-kube-api-access-kssjb\") pod \"console-operator-584dcb6b54-27xfm\" (UID: \"c0f949f6-90cf-4e28-a713-3587649d077c\") " pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:44.427799 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.427686    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0f949f6-90cf-4e28-a713-3587649d077c-serving-cert\") pod \"console-operator-584dcb6b54-27xfm\" (UID: \"c0f949f6-90cf-4e28-a713-3587649d077c\") " pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:44.427799 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.427705    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0f949f6-90cf-4e28-a713-3587649d077c-config\") pod \"console-operator-584dcb6b54-27xfm\" (UID: \"c0f949f6-90cf-4e28-a713-3587649d077c\") " pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:44.427799 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.427728    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0f949f6-90cf-4e28-a713-3587649d077c-trusted-ca\") pod \"console-operator-584dcb6b54-27xfm\" (UID: \"c0f949f6-90cf-4e28-a713-3587649d077c\") " pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:44.428432 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.428410    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0f949f6-90cf-4e28-a713-3587649d077c-config\") pod \"console-operator-584dcb6b54-27xfm\" (UID: \"c0f949f6-90cf-4e28-a713-3587649d077c\") " pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:44.428682 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.428664    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0f949f6-90cf-4e28-a713-3587649d077c-trusted-ca\") pod \"console-operator-584dcb6b54-27xfm\" (UID: \"c0f949f6-90cf-4e28-a713-3587649d077c\") " pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:44.430003 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.429977    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0f949f6-90cf-4e28-a713-3587649d077c-serving-cert\") pod \"console-operator-584dcb6b54-27xfm\" (UID: \"c0f949f6-90cf-4e28-a713-3587649d077c\") " pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:44.435382 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.435361    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kssjb\" (UniqueName: \"kubernetes.io/projected/c0f949f6-90cf-4e28-a713-3587649d077c-kube-api-access-kssjb\") pod \"console-operator-584dcb6b54-27xfm\" (UID: \"c0f949f6-90cf-4e28-a713-3587649d077c\") " pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:44.569596 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.569493    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:44.690145 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:44.690113    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-27xfm"]
Jul 29 22:50:44.693393 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:50:44.693356    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f949f6_90cf_4e28_a713_3587649d077c.slice/crio-9f9ad62e4c19acb65c45af1a09de4ba22c16ffb0cc94817155f12f60ff068bcb WatchSource:0}: Error finding container 9f9ad62e4c19acb65c45af1a09de4ba22c16ffb0cc94817155f12f60ff068bcb: Status 404 returned error can't find the container with id 9f9ad62e4c19acb65c45af1a09de4ba22c16ffb0cc94817155f12f60ff068bcb
Jul 29 22:50:45.234166 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:45.234135    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:50:45.234324 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:45.234278    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 22:50:45.234364 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:45.234337    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs podName:0d19dc00-e3da-4306-a5ff-8cca92e02a1c nodeName:}" failed. No retries permitted until 2026-07-29 22:52:47.23431961 +0000 UTC m=+374.239517023 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs") pod "network-metrics-daemon-t5x5r" (UID: "0d19dc00-e3da-4306-a5ff-8cca92e02a1c") : secret "metrics-daemon-secret" not found
Jul 29 22:50:45.243403 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:45.243370    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm" event={"ID":"c0f949f6-90cf-4e28-a713-3587649d077c","Type":"ContainerStarted","Data":"9f9ad62e4c19acb65c45af1a09de4ba22c16ffb0cc94817155f12f60ff068bcb"}
Jul 29 22:50:47.251265 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:47.251238    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/0.log"
Jul 29 22:50:47.251669 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:47.251277    2569 generic.go:356] "Generic (PLEG): container finished" podID="c0f949f6-90cf-4e28-a713-3587649d077c" containerID="512f44f179e65a7f329b0615bdf68171a92d6b5b1e0889034745dc4001424a97" exitCode=255
Jul 29 22:50:47.251669 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:47.251323    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm" event={"ID":"c0f949f6-90cf-4e28-a713-3587649d077c","Type":"ContainerDied","Data":"512f44f179e65a7f329b0615bdf68171a92d6b5b1e0889034745dc4001424a97"}
Jul 29 22:50:47.251669 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:47.251561    2569 scope.go:117] "RemoveContainer" containerID="512f44f179e65a7f329b0615bdf68171a92d6b5b1e0889034745dc4001424a97"
Jul 29 22:50:48.255869 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.255840    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/1.log"
Jul 29 22:50:48.256287 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.256181    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/0.log"
Jul 29 22:50:48.256287 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.256208    2569 generic.go:356] "Generic (PLEG): container finished" podID="c0f949f6-90cf-4e28-a713-3587649d077c" containerID="5ba24e8eaa88f27b1b97d4b8276d3c1fd34e9694376e33fdc9cd3d8902d8be24" exitCode=255
Jul 29 22:50:48.256287 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.256257    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm" event={"ID":"c0f949f6-90cf-4e28-a713-3587649d077c","Type":"ContainerDied","Data":"5ba24e8eaa88f27b1b97d4b8276d3c1fd34e9694376e33fdc9cd3d8902d8be24"}
Jul 29 22:50:48.256391 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.256307    2569 scope.go:117] "RemoveContainer" containerID="512f44f179e65a7f329b0615bdf68171a92d6b5b1e0889034745dc4001424a97"
Jul 29 22:50:48.256519 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.256478    2569 scope.go:117] "RemoveContainer" containerID="5ba24e8eaa88f27b1b97d4b8276d3c1fd34e9694376e33fdc9cd3d8902d8be24"
Jul 29 22:50:48.256701 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:48.256680    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-27xfm_openshift-console-operator(c0f949f6-90cf-4e28-a713-3587649d077c)\"" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm" podUID="c0f949f6-90cf-4e28-a713-3587649d077c"
Jul 29 22:50:48.737073 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.737036    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5d2kg"]
Jul 29 22:50:48.741224 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.741202    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:48.742983 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.742955    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 22:50:48.743236 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.743222    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gfc6t\""
Jul 29 22:50:48.743685 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.743665    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 22:50:48.743685 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.743669    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 22:50:48.743875 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.743699    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 22:50:48.750397 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.750376    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5d2kg"]
Jul 29 22:50:48.859037 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.859002    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bf23056-3f7e-414c-917a-1261f7310bdb-data-volume\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:48.859211 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.859042    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdr2b\" (UniqueName: \"kubernetes.io/projected/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-api-access-wdr2b\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:48.859211 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.859121    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:48.859211 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.859145    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:48.859211 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.859172    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1bf23056-3f7e-414c-917a-1261f7310bdb-crio-socket\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:48.960496 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.960451    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1bf23056-3f7e-414c-917a-1261f7310bdb-crio-socket\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:48.960582 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.960522    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bf23056-3f7e-414c-917a-1261f7310bdb-data-volume\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:48.960582 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.960542    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdr2b\" (UniqueName: \"kubernetes.io/projected/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-api-access-wdr2b\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:48.960654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.960585    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1bf23056-3f7e-414c-917a-1261f7310bdb-crio-socket\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:48.960654 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.960596    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:48.960739 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.960658    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:48.960785 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:48.960750    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 22:50:48.960785 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:48.960766    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:48.960878 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:48.960835    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm podName:1bf23056-3f7e-414c-917a-1261f7310bdb nodeName:}" failed. No retries permitted until 2026-07-29 22:50:49.460812825 +0000 UTC m=+256.466010243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5d2kg" (UID: "1bf23056-3f7e-414c-917a-1261f7310bdb") : configmap "kube-rbac-proxy" not found
Jul 29 22:50:48.960878 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:48.960858    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls podName:1bf23056-3f7e-414c-917a-1261f7310bdb nodeName:}" failed. No retries permitted until 2026-07-29 22:50:49.460847889 +0000 UTC m=+256.466045291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5d2kg" (UID: "1bf23056-3f7e-414c-917a-1261f7310bdb") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:48.960966 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.960912    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bf23056-3f7e-414c-917a-1261f7310bdb-data-volume\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:48.968709 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:48.968687    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdr2b\" (UniqueName: \"kubernetes.io/projected/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-api-access-wdr2b\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:49.260204 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:49.260172    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/1.log"
Jul 29 22:50:49.260607 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:49.260471    2569 scope.go:117] "RemoveContainer" containerID="5ba24e8eaa88f27b1b97d4b8276d3c1fd34e9694376e33fdc9cd3d8902d8be24"
Jul 29 22:50:49.260657 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:49.260638    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-27xfm_openshift-console-operator(c0f949f6-90cf-4e28-a713-3587649d077c)\"" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm" podUID="c0f949f6-90cf-4e28-a713-3587649d077c"
Jul 29 22:50:49.465552 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:49.465512    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:49.465552 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:49.465559    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:49.465783 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:49.465621    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 22:50:49.465783 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:49.465687    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm podName:1bf23056-3f7e-414c-917a-1261f7310bdb nodeName:}" failed. No retries permitted until 2026-07-29 22:50:50.465671294 +0000 UTC m=+257.470868698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5d2kg" (UID: "1bf23056-3f7e-414c-917a-1261f7310bdb") : configmap "kube-rbac-proxy" not found
Jul 29 22:50:49.465783 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:49.465695    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:49.465783 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:49.465754    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls podName:1bf23056-3f7e-414c-917a-1261f7310bdb nodeName:}" failed. No retries permitted until 2026-07-29 22:50:50.465742903 +0000 UTC m=+257.470940305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5d2kg" (UID: "1bf23056-3f7e-414c-917a-1261f7310bdb") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:50.472919 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:50.472862    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:50.472919 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:50.472920    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:50.473337 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:50.472985    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 22:50:50.473337 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:50.473021    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:50.473337 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:50.473050    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm podName:1bf23056-3f7e-414c-917a-1261f7310bdb nodeName:}" failed. No retries permitted until 2026-07-29 22:50:52.473035622 +0000 UTC m=+259.478233023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5d2kg" (UID: "1bf23056-3f7e-414c-917a-1261f7310bdb") : configmap "kube-rbac-proxy" not found
Jul 29 22:50:50.473337 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:50.473065    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls podName:1bf23056-3f7e-414c-917a-1261f7310bdb nodeName:}" failed. No retries permitted until 2026-07-29 22:50:52.473058372 +0000 UTC m=+259.478255772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5d2kg" (UID: "1bf23056-3f7e-414c-917a-1261f7310bdb") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:51.341624 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:51.341596    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mbdtd_06f404ba-6594-4814-b5c5-9904a205207c/dns-node-resolver/0.log"
Jul 29 22:50:52.489476 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:52.489436    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:52.489879 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:52.489503    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:52.489879 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:52.489556    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 22:50:52.489879 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:52.489631    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm podName:1bf23056-3f7e-414c-917a-1261f7310bdb nodeName:}" failed. No retries permitted until 2026-07-29 22:50:56.489609689 +0000 UTC m=+263.494807093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5d2kg" (UID: "1bf23056-3f7e-414c-917a-1261f7310bdb") : configmap "kube-rbac-proxy" not found
Jul 29 22:50:52.489879 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:52.489650    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:52.489879 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:52.489689    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls podName:1bf23056-3f7e-414c-917a-1261f7310bdb nodeName:}" failed. No retries permitted until 2026-07-29 22:50:56.489677966 +0000 UTC m=+263.494875373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5d2kg" (UID: "1bf23056-3f7e-414c-917a-1261f7310bdb") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:52.741966 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:52.741886    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r8cns_8402f600-296c-4fa7-a6a2-1d34bb23781c/node-ca/0.log"
Jul 29 22:50:53.542026 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:53.541964    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wf4k2_bc372877-f198-4771-b46c-8ebd74cca4bc/serve-healthcheck-canary/0.log"
Jul 29 22:50:54.570500 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:54.570395    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:54.570500 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:54.570497    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:50:54.570890 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:54.570817    2569 scope.go:117] "RemoveContainer" containerID="5ba24e8eaa88f27b1b97d4b8276d3c1fd34e9694376e33fdc9cd3d8902d8be24"
Jul 29 22:50:54.570993 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:54.570972    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-27xfm_openshift-console-operator(c0f949f6-90cf-4e28-a713-3587649d077c)\"" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm" podUID="c0f949f6-90cf-4e28-a713-3587649d077c"
Jul 29 22:50:56.517625 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:56.517535    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:56.517625 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:50:56.517577    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:50:56.518007 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:56.517675    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 22:50:56.518007 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:56.517763    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm podName:1bf23056-3f7e-414c-917a-1261f7310bdb nodeName:}" failed. No retries permitted until 2026-07-29 22:51:04.517745717 +0000 UTC m=+271.522943120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5d2kg" (UID: "1bf23056-3f7e-414c-917a-1261f7310bdb") : configmap "kube-rbac-proxy" not found
Jul 29 22:50:56.518007 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:56.517686    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:56.518007 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:50:56.517830    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls podName:1bf23056-3f7e-414c-917a-1261f7310bdb nodeName:}" failed. No retries permitted until 2026-07-29 22:51:04.517816893 +0000 UTC m=+271.523014294 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5d2kg" (UID: "1bf23056-3f7e-414c-917a-1261f7310bdb") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:51:04.579859 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:04.579801    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:51:04.579859 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:04.579860    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:51:04.580291 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:51:04.579957    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:51:04.580291 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:51:04.580013    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls podName:1bf23056-3f7e-414c-917a-1261f7310bdb nodeName:}" failed. No retries permitted until 2026-07-29 22:51:20.579997292 +0000 UTC m=+287.585194695 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5d2kg" (UID: "1bf23056-3f7e-414c-917a-1261f7310bdb") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:51:04.580367 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:04.580305    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bf23056-3f7e-414c-917a-1261f7310bdb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:51:06.469300 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:06.469270    2569 scope.go:117] "RemoveContainer" containerID="5ba24e8eaa88f27b1b97d4b8276d3c1fd34e9694376e33fdc9cd3d8902d8be24"
Jul 29 22:51:07.311261 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:07.311233    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 22:51:07.311613 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:07.311597    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/1.log"
Jul 29 22:51:07.311676 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:07.311629    2569 generic.go:356] "Generic (PLEG): container finished" podID="c0f949f6-90cf-4e28-a713-3587649d077c" containerID="2b0a0c0287e0be5e9523a9fdf440e936987765ab8d23309e4570b02f1d8a277f" exitCode=255
Jul 29 22:51:07.311725 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:07.311678    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm" event={"ID":"c0f949f6-90cf-4e28-a713-3587649d077c","Type":"ContainerDied","Data":"2b0a0c0287e0be5e9523a9fdf440e936987765ab8d23309e4570b02f1d8a277f"}
Jul 29 22:51:07.311774 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:07.311725    2569 scope.go:117] "RemoveContainer" containerID="5ba24e8eaa88f27b1b97d4b8276d3c1fd34e9694376e33fdc9cd3d8902d8be24"
Jul 29 22:51:07.312037 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:07.312020    2569 scope.go:117] "RemoveContainer" containerID="2b0a0c0287e0be5e9523a9fdf440e936987765ab8d23309e4570b02f1d8a277f"
Jul 29 22:51:07.312239 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:51:07.312204    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-27xfm_openshift-console-operator(c0f949f6-90cf-4e28-a713-3587649d077c)\"" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm" podUID="c0f949f6-90cf-4e28-a713-3587649d077c"
Jul 29 22:51:08.316431 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:08.316401    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 22:51:13.006689 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.006646    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7778c94ff4-8ns4j"]
Jul 29 22:51:13.011418 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.011401    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.014148 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.014118    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vpnhm\""
Jul 29 22:51:13.014295 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.014165    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 22:51:13.014579 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.014557    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 22:51:13.014686 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.014585    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 22:51:13.019948 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.019930    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 22:51:13.028186 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.028164    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7778c94ff4-8ns4j"]
Jul 29 22:51:13.041081 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.041052    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-bound-sa-token\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.041211 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.041099    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-trusted-ca\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.041211 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.041119    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-ca-trust-extracted\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.041211 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.041160    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-image-registry-private-configuration\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.041309 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.041228    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-installation-pull-secrets\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.041309 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.041273    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-registry-certificates\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.041309 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.041294    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-registry-tls\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.041394 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.041313    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9dzh\" (UniqueName: \"kubernetes.io/projected/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-kube-api-access-f9dzh\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.141637 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.141601    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-registry-certificates\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.141637 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.141639    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-registry-tls\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.141894 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.141656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9dzh\" (UniqueName: \"kubernetes.io/projected/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-kube-api-access-f9dzh\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.141894 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.141690    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-bound-sa-token\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.141894 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.141717    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-trusted-ca\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.141894 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.141734    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-ca-trust-extracted\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.141894 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.141756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-image-registry-private-configuration\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.141894 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.141803    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-installation-pull-secrets\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.142317 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.142293    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-ca-trust-extracted\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.142648 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.142627    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-registry-certificates\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.142924 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.142900    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-trusted-ca\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.144089 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.144064    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-registry-tls\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.144179 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.144142    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-image-registry-private-configuration\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.144266 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.144250    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-installation-pull-secrets\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.152521 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.152476    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-bound-sa-token\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.152744 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.152725    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9dzh\" (UniqueName: \"kubernetes.io/projected/bbd2381d-ee5b-4a80-9d71-a98c3866ed4f-kube-api-access-f9dzh\") pod \"image-registry-7778c94ff4-8ns4j\" (UID: \"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f\") " pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.319915 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.319823    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:13.444258 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:13.444223    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7778c94ff4-8ns4j"]
Jul 29 22:51:13.448057 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:51:13.448030    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd2381d_ee5b_4a80_9d71_a98c3866ed4f.slice/crio-ef871df21f4cd814d41c86f1f8fc2c56a56290851d13b17d36ae6b8629721c2f WatchSource:0}: Error finding container ef871df21f4cd814d41c86f1f8fc2c56a56290851d13b17d36ae6b8629721c2f: Status 404 returned error can't find the container with id ef871df21f4cd814d41c86f1f8fc2c56a56290851d13b17d36ae6b8629721c2f
Jul 29 22:51:14.338595 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:14.338553    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j" event={"ID":"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f","Type":"ContainerStarted","Data":"d12276c456ddb5fd64fce2bb4fed169dd3b823155c35fd2225672b63fb7b2139"}
Jul 29 22:51:14.339013 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:14.338649    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j" event={"ID":"bbd2381d-ee5b-4a80-9d71-a98c3866ed4f","Type":"ContainerStarted","Data":"ef871df21f4cd814d41c86f1f8fc2c56a56290851d13b17d36ae6b8629721c2f"}
Jul 29 22:51:14.339013 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:14.338679    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:14.357199 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:14.357150    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j" podStartSLOduration=2.35713426 podStartE2EDuration="2.35713426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:14.355802766 +0000 UTC m=+281.361000200" watchObservedRunningTime="2026-07-29 22:51:14.35713426 +0000 UTC m=+281.362331712"
Jul 29 22:51:14.569920 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:14.569882    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:51:14.570085 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:14.569946    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:51:14.570302 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:14.570287    2569 scope.go:117] "RemoveContainer" containerID="2b0a0c0287e0be5e9523a9fdf440e936987765ab8d23309e4570b02f1d8a277f"
Jul 29 22:51:14.570504 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:51:14.570461    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-27xfm_openshift-console-operator(c0f949f6-90cf-4e28-a713-3587649d077c)\"" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm" podUID="c0f949f6-90cf-4e28-a713-3587649d077c"
Jul 29 22:51:14.991595 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:51:14.991542    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xxsch" podUID="49ebc216-8a22-460c-b045-06da7890eba1"
Jul 29 22:51:15.341042 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:15.340967    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxsch"
Jul 29 22:51:18.882700 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:18.882663    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:51:18.884893 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:18.884873    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ebc216-8a22-460c-b045-06da7890eba1-metrics-tls\") pod \"dns-default-xxsch\" (UID: \"49ebc216-8a22-460c-b045-06da7890eba1\") " pod="openshift-dns/dns-default-xxsch"
Jul 29 22:51:18.943957 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:18.943923    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jtttq\""
Jul 29 22:51:18.952558 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:18.952532    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxsch"
Jul 29 22:51:19.077809 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:19.077776    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xxsch"]
Jul 29 22:51:19.083009 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:51:19.082962    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ebc216_8a22_460c_b045_06da7890eba1.slice/crio-6f0dae6a1b79053113b363b46f5e49045290e85e59d7914fe8ae16b46f3a5c84 WatchSource:0}: Error finding container 6f0dae6a1b79053113b363b46f5e49045290e85e59d7914fe8ae16b46f3a5c84: Status 404 returned error can't find the container with id 6f0dae6a1b79053113b363b46f5e49045290e85e59d7914fe8ae16b46f3a5c84
Jul 29 22:51:19.354106 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:19.354072    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxsch" event={"ID":"49ebc216-8a22-460c-b045-06da7890eba1","Type":"ContainerStarted","Data":"6f0dae6a1b79053113b363b46f5e49045290e85e59d7914fe8ae16b46f3a5c84"}
Jul 29 22:51:20.596377 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:20.596337    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:51:20.598616 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:20.598589    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bf23056-3f7e-414c-917a-1261f7310bdb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d2kg\" (UID: \"1bf23056-3f7e-414c-917a-1261f7310bdb\") " pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:51:20.850140 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:20.850050    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5d2kg"
Jul 29 22:51:20.975292 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:20.975258    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5d2kg"]
Jul 29 22:51:20.978410 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:51:20.978388    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf23056_3f7e_414c_917a_1261f7310bdb.slice/crio-b3d857f75299ea83831e73ae0d2ba5e1d573e48e1f834e12195b10b2695ca563 WatchSource:0}: Error finding container b3d857f75299ea83831e73ae0d2ba5e1d573e48e1f834e12195b10b2695ca563: Status 404 returned error can't find the container with id b3d857f75299ea83831e73ae0d2ba5e1d573e48e1f834e12195b10b2695ca563
Jul 29 22:51:21.364152 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:21.364101    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5d2kg" event={"ID":"1bf23056-3f7e-414c-917a-1261f7310bdb","Type":"ContainerStarted","Data":"b3d857f75299ea83831e73ae0d2ba5e1d573e48e1f834e12195b10b2695ca563"}
Jul 29 22:51:21.366342 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:21.366308    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxsch" event={"ID":"49ebc216-8a22-460c-b045-06da7890eba1","Type":"ContainerStarted","Data":"1117b8ca4d506fc77028ce68f6f60e1d7b463baa299b2429a12528a3e13d5c32"}
Jul 29 22:51:21.366506 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:21.366369    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxsch" event={"ID":"49ebc216-8a22-460c-b045-06da7890eba1","Type":"ContainerStarted","Data":"51cbdf2420a5af8d6040216b6ff36a601274b3ca843d3fd95256dd3820ac8e4e"}
Jul 29 22:51:21.366572 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:21.366503    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xxsch"
Jul 29 22:51:21.389396 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:21.389324    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xxsch" podStartSLOduration=251.862792844 podStartE2EDuration="4m13.389299405s" totalImagesPullingTime="1.526506561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:21.383101329 +0000 UTC m=+288.388298760" watchObservedRunningTime="2026-07-29 22:51:21.389299405 +0000 UTC m=+288.394496829"
Jul 29 22:51:26.386826 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:26.386742    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5d2kg" event={"ID":"1bf23056-3f7e-414c-917a-1261f7310bdb","Type":"ContainerStarted","Data":"c219d7cf66330c7d5d1d95eaee242124816b10b48506b47dd8653e9cf2a9038c"}
Jul 29 22:51:29.397773 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:29.397716    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5d2kg" event={"ID":"1bf23056-3f7e-414c-917a-1261f7310bdb","Type":"ContainerStarted","Data":"14f1157f2e1ab301e0b01b825d6db0a3acafbedc5f1b8b4782ac38606ca1bdc0"}
Jul 29 22:51:29.469628 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:29.469598    2569 scope.go:117] "RemoveContainer" containerID="2b0a0c0287e0be5e9523a9fdf440e936987765ab8d23309e4570b02f1d8a277f"
Jul 29 22:51:30.112704 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.112664    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-4qmvc"]
Jul 29 22:51:30.115840 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.115816    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-4qmvc"
Jul 29 22:51:30.117980 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.117954    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 22:51:30.118270 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.118239    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 22:51:30.118387 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.118291    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-9kgn2\""
Jul 29 22:51:30.126309 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.126276    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-4qmvc"]
Jul 29 22:51:30.177497 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.177452    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49wtp\" (UniqueName: \"kubernetes.io/projected/5e2c4d0c-99fa-45cd-b046-82a40d8a97a0-kube-api-access-49wtp\") pod \"downloads-7c97dc4555-4qmvc\" (UID: \"5e2c4d0c-99fa-45cd-b046-82a40d8a97a0\") " pod="openshift-console/downloads-7c97dc4555-4qmvc"
Jul 29 22:51:30.278580 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.278546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49wtp\" (UniqueName: \"kubernetes.io/projected/5e2c4d0c-99fa-45cd-b046-82a40d8a97a0-kube-api-access-49wtp\") pod \"downloads-7c97dc4555-4qmvc\" (UID: \"5e2c4d0c-99fa-45cd-b046-82a40d8a97a0\") " pod="openshift-console/downloads-7c97dc4555-4qmvc"
Jul 29 22:51:30.290538 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.290507    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49wtp\" (UniqueName: \"kubernetes.io/projected/5e2c4d0c-99fa-45cd-b046-82a40d8a97a0-kube-api-access-49wtp\") pod \"downloads-7c97dc4555-4qmvc\" (UID: \"5e2c4d0c-99fa-45cd-b046-82a40d8a97a0\") " pod="openshift-console/downloads-7c97dc4555-4qmvc"
Jul 29 22:51:30.403409 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.403329    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 22:51:30.403409 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.403393    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm" event={"ID":"c0f949f6-90cf-4e28-a713-3587649d077c","Type":"ContainerStarted","Data":"81e6e4a546a8c00f226d88fd180ad6511f50778838975666f6c831ae6166743d"}
Jul 29 22:51:30.403898 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.403677    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:51:30.409433 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.409408    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm"
Jul 29 22:51:30.421849 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.421788    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-27xfm" podStartSLOduration=44.516958922 podStartE2EDuration="46.421767826s" totalImagesPullingTime="1.904808904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:30.420858231 +0000 UTC m=+297.426055657" watchObservedRunningTime="2026-07-29 22:51:30.421767826 +0000 UTC m=+297.426965461"
Jul 29 22:51:30.426449 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.426424    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-4qmvc"
Jul 29 22:51:30.469871 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:51:30.469813    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-t5x5r" podUID="0d19dc00-e3da-4306-a5ff-8cca92e02a1c"
Jul 29 22:51:30.574314 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:30.574272    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-4qmvc"]
Jul 29 22:51:30.580267 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:51:30.580196    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2c4d0c_99fa_45cd_b046_82a40d8a97a0.slice/crio-49d2668e7d01e826118cdadc5153c535964cadab4a20f88b246c2f008b22366f WatchSource:0}: Error finding container 49d2668e7d01e826118cdadc5153c535964cadab4a20f88b246c2f008b22366f: Status 404 returned error can't find the container with id 49d2668e7d01e826118cdadc5153c535964cadab4a20f88b246c2f008b22366f
Jul 29 22:51:31.373429 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.373400    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xxsch"
Jul 29 22:51:31.410691 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.410599    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-4qmvc" event={"ID":"5e2c4d0c-99fa-45cd-b046-82a40d8a97a0","Type":"ContainerStarted","Data":"49d2668e7d01e826118cdadc5153c535964cadab4a20f88b246c2f008b22366f"}
Jul 29 22:51:31.579513 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.579345    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"]
Jul 29 22:51:31.585849 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.585162    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.589411 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.588108    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kn7nm"]
Jul 29 22:51:31.589411 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.589267    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 29 22:51:31.589632 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.589419    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 22:51:31.589632 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.589501    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-d89bl\""
Jul 29 22:51:31.590904 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.590874    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 29 22:51:31.591115 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.591098    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 29 22:51:31.591430 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.591409    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 22:51:31.591430 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.591426    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.591692 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.591674    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 22:51:31.594805 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.594783    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 22:51:31.599666 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.595953    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 22:51:31.599666 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.596221    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 22:51:31.599666 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.596463    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xxcw6\""
Jul 29 22:51:31.599666 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.596584    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"]
Jul 29 22:51:31.690581 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.690496    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/704b9290-51eb-4000-9ab2-7ae30d6f8764-node-exporter-wtmp\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.690581 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.690547    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs8s4\" (UniqueName: \"kubernetes.io/projected/704b9290-51eb-4000-9ab2-7ae30d6f8764-kube-api-access-qs8s4\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.690812 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.690611    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/704b9290-51eb-4000-9ab2-7ae30d6f8764-metrics-client-ca\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.690812 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.690656    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8088e072-67e5-4ff4-a931-e881cb0e69fa-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.690812 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.690681    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/704b9290-51eb-4000-9ab2-7ae30d6f8764-sys\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.690812 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.690751    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8088e072-67e5-4ff4-a931-e881cb0e69fa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.690812 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.690799    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8088e072-67e5-4ff4-a931-e881cb0e69fa-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.691082 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.690832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8088e072-67e5-4ff4-a931-e881cb0e69fa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.691082 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.690861    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/704b9290-51eb-4000-9ab2-7ae30d6f8764-node-exporter-tls\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.691082 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.690935    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/704b9290-51eb-4000-9ab2-7ae30d6f8764-root\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.691082 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.690963    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkm65\" (UniqueName: \"kubernetes.io/projected/8088e072-67e5-4ff4-a931-e881cb0e69fa-kube-api-access-kkm65\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.691082 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.690986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/704b9290-51eb-4000-9ab2-7ae30d6f8764-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.691082 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.691009    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/704b9290-51eb-4000-9ab2-7ae30d6f8764-node-exporter-textfile\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.691082 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.691037    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8088e072-67e5-4ff4-a931-e881cb0e69fa-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.691082 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.691063    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/704b9290-51eb-4000-9ab2-7ae30d6f8764-node-exporter-accelerators-collector-config\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.791852 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.791812    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8088e072-67e5-4ff4-a931-e881cb0e69fa-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.791852 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.791859    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/704b9290-51eb-4000-9ab2-7ae30d6f8764-sys\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.792095 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.791901    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8088e072-67e5-4ff4-a931-e881cb0e69fa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.792095 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.791935    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8088e072-67e5-4ff4-a931-e881cb0e69fa-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.792095 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.791970    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8088e072-67e5-4ff4-a931-e881cb0e69fa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.792095 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.791997    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/704b9290-51eb-4000-9ab2-7ae30d6f8764-node-exporter-tls\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.792095 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.792055    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/704b9290-51eb-4000-9ab2-7ae30d6f8764-root\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.792095 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.792082    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkm65\" (UniqueName: \"kubernetes.io/projected/8088e072-67e5-4ff4-a931-e881cb0e69fa-kube-api-access-kkm65\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.792095 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:51:31.792087    2569 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 29 22:51:31.792427 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.792107    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/704b9290-51eb-4000-9ab2-7ae30d6f8764-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.792427 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.792134    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/704b9290-51eb-4000-9ab2-7ae30d6f8764-node-exporter-textfile\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.792427 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:51:31.792163    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8088e072-67e5-4ff4-a931-e881cb0e69fa-kube-state-metrics-tls podName:8088e072-67e5-4ff4-a931-e881cb0e69fa nodeName:}" failed. No retries permitted until 2026-07-29 22:51:32.292141949 +0000 UTC m=+299.297339369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8088e072-67e5-4ff4-a931-e881cb0e69fa-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-lzctb" (UID: "8088e072-67e5-4ff4-a931-e881cb0e69fa") : secret "kube-state-metrics-tls" not found
Jul 29 22:51:31.792427 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.792204    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8088e072-67e5-4ff4-a931-e881cb0e69fa-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.792427 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.792234    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/704b9290-51eb-4000-9ab2-7ae30d6f8764-node-exporter-accelerators-collector-config\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.792427 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.792282    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/704b9290-51eb-4000-9ab2-7ae30d6f8764-node-exporter-wtmp\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.792427 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.792307    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs8s4\" (UniqueName: \"kubernetes.io/projected/704b9290-51eb-4000-9ab2-7ae30d6f8764-kube-api-access-qs8s4\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.792427 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.792340    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/704b9290-51eb-4000-9ab2-7ae30d6f8764-metrics-client-ca\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.792427 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.792425    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/704b9290-51eb-4000-9ab2-7ae30d6f8764-node-exporter-textfile\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.792896 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.792662    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/704b9290-51eb-4000-9ab2-7ae30d6f8764-sys\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.792896 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.792723    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/704b9290-51eb-4000-9ab2-7ae30d6f8764-root\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.792896 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.792794    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8088e072-67e5-4ff4-a931-e881cb0e69fa-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.793044 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.793008    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/704b9290-51eb-4000-9ab2-7ae30d6f8764-metrics-client-ca\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.793100 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.793066    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8088e072-67e5-4ff4-a931-e881cb0e69fa-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.793177 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.793103    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/704b9290-51eb-4000-9ab2-7ae30d6f8764-node-exporter-wtmp\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.793319 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.793295    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8088e072-67e5-4ff4-a931-e881cb0e69fa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.793519 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.793473    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/704b9290-51eb-4000-9ab2-7ae30d6f8764-node-exporter-accelerators-collector-config\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.796358 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.795927    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/704b9290-51eb-4000-9ab2-7ae30d6f8764-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.796358 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.796322    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8088e072-67e5-4ff4-a931-e881cb0e69fa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.799188 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.799143    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/704b9290-51eb-4000-9ab2-7ae30d6f8764-node-exporter-tls\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.804724 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.804664    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkm65\" (UniqueName: \"kubernetes.io/projected/8088e072-67e5-4ff4-a931-e881cb0e69fa-kube-api-access-kkm65\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:31.805251 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.805204    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs8s4\" (UniqueName: \"kubernetes.io/projected/704b9290-51eb-4000-9ab2-7ae30d6f8764-kube-api-access-qs8s4\") pod \"node-exporter-kn7nm\" (UID: \"704b9290-51eb-4000-9ab2-7ae30d6f8764\") " pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:31.911908 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:31.911868    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kn7nm"
Jul 29 22:51:32.298747 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:32.298698    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8088e072-67e5-4ff4-a931-e881cb0e69fa-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:32.301587 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:32.301545    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8088e072-67e5-4ff4-a931-e881cb0e69fa-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lzctb\" (UID: \"8088e072-67e5-4ff4-a931-e881cb0e69fa\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:32.503107 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:32.503070    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"
Jul 29 22:51:33.392905 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:51:33.392852    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704b9290_51eb_4000_9ab2_7ae30d6f8764.slice/crio-81d9c7b565946e35c8b88f46abd3fb5a00eba7b6b405251fb5f339d9ebc5b9a6 WatchSource:0}: Error finding container 81d9c7b565946e35c8b88f46abd3fb5a00eba7b6b405251fb5f339d9ebc5b9a6: Status 404 returned error can't find the container with id 81d9c7b565946e35c8b88f46abd3fb5a00eba7b6b405251fb5f339d9ebc5b9a6
Jul 29 22:51:33.402218 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:33.402192    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 22:51:33.402448 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:33.402429    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 22:51:33.419718 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:33.419683    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kn7nm" event={"ID":"704b9290-51eb-4000-9ab2-7ae30d6f8764","Type":"ContainerStarted","Data":"81d9c7b565946e35c8b88f46abd3fb5a00eba7b6b405251fb5f339d9ebc5b9a6"}
Jul 29 22:51:33.420121 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:33.420101    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 22:51:33.539291 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:33.539257    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb"]
Jul 29 22:51:33.542380 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:51:33.542350    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8088e072_67e5_4ff4_a931_e881cb0e69fa.slice/crio-1132492e025e2df55fe2f682a6527c8eee43e924ca778dcbf1373bbf25ee01db WatchSource:0}: Error finding container 1132492e025e2df55fe2f682a6527c8eee43e924ca778dcbf1373bbf25ee01db: Status 404 returned error can't find the container with id 1132492e025e2df55fe2f682a6527c8eee43e924ca778dcbf1373bbf25ee01db
Jul 29 22:51:34.424833 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:34.424789    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb" event={"ID":"8088e072-67e5-4ff4-a931-e881cb0e69fa","Type":"ContainerStarted","Data":"1132492e025e2df55fe2f682a6527c8eee43e924ca778dcbf1373bbf25ee01db"}
Jul 29 22:51:34.432000 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:34.431945    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5d2kg" event={"ID":"1bf23056-3f7e-414c-917a-1261f7310bdb","Type":"ContainerStarted","Data":"b7bfed74ab5ef49b08a0cbb33cceaf6f35b02ac7e2371e25398d965e697c4a09"}
Jul 29 22:51:34.457760 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:34.457701    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5d2kg" podStartSLOduration=34.101139128 podStartE2EDuration="46.457681461s" totalImagesPullingTime="12.356542333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:50:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:34.456539266 +0000 UTC m=+301.461736705" watchObservedRunningTime="2026-07-29 22:51:34.457681461 +0000 UTC m=+301.462878886"
Jul 29 22:51:35.346961 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.346929    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7778c94ff4-8ns4j"
Jul 29 22:51:35.437979 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.437944    2569 generic.go:356] "Generic (PLEG): container finished" podID="704b9290-51eb-4000-9ab2-7ae30d6f8764" containerID="2e3461efbcdb888ea7881245d6d88163bbd8e5859e1e5ac6999c0f79f148fc31" exitCode=0
Jul 29 22:51:35.438158 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.438031    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kn7nm" event={"ID":"704b9290-51eb-4000-9ab2-7ae30d6f8764","Type":"ContainerDied","Data":"2e3461efbcdb888ea7881245d6d88163bbd8e5859e1e5ac6999c0f79f148fc31"}
Jul 29 22:51:35.440955 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.440815    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb" event={"ID":"8088e072-67e5-4ff4-a931-e881cb0e69fa","Type":"ContainerStarted","Data":"102c2e0c677f8eb19cdffbf36f6495042343c91185fb2ac57e8fdd937c04efe2"}
Jul 29 22:51:35.440955 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.440875    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb" event={"ID":"8088e072-67e5-4ff4-a931-e881cb0e69fa","Type":"ContainerStarted","Data":"c37377a158a01ae1dda2046711433bbca7b369a999f2c6a16a0f9187cc75d141"}
Jul 29 22:51:35.440955 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.440896    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb" event={"ID":"8088e072-67e5-4ff4-a931-e881cb0e69fa","Type":"ContainerStarted","Data":"fcaeae200f2c5e46659dcb3859bc72088f139386c23acb09c0886f367247b072"}
Jul 29 22:51:35.481513 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.481446    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lzctb" podStartSLOduration=3.021555498 podStartE2EDuration="4.481430302s" totalImagesPullingTime="1.459874804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:35.480436703 +0000 UTC m=+302.485634127" watchObservedRunningTime="2026-07-29 22:51:35.481430302 +0000 UTC m=+302.486627752"
Jul 29 22:51:35.925941 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.925860    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7bb87447bb-z4kjc"]
Jul 29 22:51:35.929131 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.929106    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:35.931305 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.931262    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 29 22:51:35.931305 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.931299    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 29 22:51:35.931516 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.931497    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-acrqkmsvaofhc\""
Jul 29 22:51:35.931623 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.931499    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 22:51:35.931623 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.931595    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 29 22:51:35.931730 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.931640    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8hf74\""
Jul 29 22:51:35.940140 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.940114    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bb87447bb-z4kjc"]
Jul 29 22:51:35.993220 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:35.993175    2569 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 29 22:51:36.035086 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.035051    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n8jm\" (UniqueName: \"kubernetes.io/projected/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-kube-api-access-2n8jm\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.035262 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.035102    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-audit-log\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.035262 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.035155    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-client-ca-bundle\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.035262 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.035183    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-metrics-server-audit-profiles\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.035262 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.035214    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.035262 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.035245    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-secret-metrics-server-client-certs\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.035535 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.035279    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-secret-metrics-server-tls\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.136374 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.136333    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-client-ca-bundle\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.136598 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.136392    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-metrics-server-audit-profiles\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.136598 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.136441    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.136598 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.136477    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-secret-metrics-server-client-certs\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.136598 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.136538    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-secret-metrics-server-tls\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.136797 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.136617    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n8jm\" (UniqueName: \"kubernetes.io/projected/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-kube-api-access-2n8jm\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.136797 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.136645    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-audit-log\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.137113 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.137056    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-audit-log\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.137756 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.137331    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.137888 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.137777    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-metrics-server-audit-profiles\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.139533 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.139509    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-client-ca-bundle\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.139737 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.139696    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-secret-metrics-server-client-certs\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.139737 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.139550    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-secret-metrics-server-tls\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.145270 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.145211    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n8jm\" (UniqueName: \"kubernetes.io/projected/275a9a4a-e06e-40a9-9c9e-45562d22fcbd-kube-api-access-2n8jm\") pod \"metrics-server-7bb87447bb-z4kjc\" (UID: \"275a9a4a-e06e-40a9-9c9e-45562d22fcbd\") " pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.241113 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.241067    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:36.387808 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.387762    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bb87447bb-z4kjc"]
Jul 29 22:51:36.390868 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:51:36.390832    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275a9a4a_e06e_40a9_9c9e_45562d22fcbd.slice/crio-e1adf2a96b69c4d5dae75339fd792b134dc28552a31ae49777e8ed18f8b24592 WatchSource:0}: Error finding container e1adf2a96b69c4d5dae75339fd792b134dc28552a31ae49777e8ed18f8b24592: Status 404 returned error can't find the container with id e1adf2a96b69c4d5dae75339fd792b134dc28552a31ae49777e8ed18f8b24592
Jul 29 22:51:36.393150 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.393133    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 22:51:36.446672 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.446620    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kn7nm" event={"ID":"704b9290-51eb-4000-9ab2-7ae30d6f8764","Type":"ContainerStarted","Data":"0c124ad6b075482e9396d25aaa7c47ed04df822043f3ec0fbf7d1fde43aa761a"}
Jul 29 22:51:36.446857 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.446683    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kn7nm" event={"ID":"704b9290-51eb-4000-9ab2-7ae30d6f8764","Type":"ContainerStarted","Data":"4ef204e2f3d15dc564572e3d77efa346b81ee8166c8b00b2fd7a7746891a29fc"}
Jul 29 22:51:36.448062 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.448026    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc" event={"ID":"275a9a4a-e06e-40a9-9c9e-45562d22fcbd","Type":"ContainerStarted","Data":"e1adf2a96b69c4d5dae75339fd792b134dc28552a31ae49777e8ed18f8b24592"}
Jul 29 22:51:36.467086 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.467025    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kn7nm" podStartSLOduration=4.347677937 podStartE2EDuration="5.467003514s" totalImagesPullingTime="1.04342887s" totalInitContainerRuntime="75.896707ms" isStatefulPod=true podCreationTimestamp="2026-07-29 22:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:36.465845195 +0000 UTC m=+303.471042619" watchObservedRunningTime="2026-07-29 22:51:36.467003514 +0000 UTC m=+303.472200941"
Jul 29 22:51:36.701340 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.701263    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5df795b4f-bxd4s"]
Jul 29 22:51:36.705746 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.705727    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.708006 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.707973    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 29 22:51:36.708194 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.708078    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 29 22:51:36.708311 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.708298    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fnp45\""
Jul 29 22:51:36.708614 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.708596    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 29 22:51:36.708767 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.708754    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 29 22:51:36.709017 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.709001    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 29 22:51:36.715773 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.715750    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 29 22:51:36.719284 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.719262    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5df795b4f-bxd4s"]
Jul 29 22:51:36.844698 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.844660    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35208e50-6550-46e0-8c9d-6e25ddf655ff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.844865 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.844711    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35208e50-6550-46e0-8c9d-6e25ddf655ff-metrics-client-ca\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.844865 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.844769    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/35208e50-6550-46e0-8c9d-6e25ddf655ff-telemeter-client-tls\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.844865 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.844809    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35208e50-6550-46e0-8c9d-6e25ddf655ff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.844996 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.844872    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35208e50-6550-46e0-8c9d-6e25ddf655ff-serving-certs-ca-bundle\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.844996 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.844930    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/35208e50-6550-46e0-8c9d-6e25ddf655ff-federate-client-tls\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.844996 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.844967    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrxt\" (UniqueName: \"kubernetes.io/projected/35208e50-6550-46e0-8c9d-6e25ddf655ff-kube-api-access-rwrxt\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.845120 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.845009    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/35208e50-6550-46e0-8c9d-6e25ddf655ff-secret-telemeter-client\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.945895 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.945861    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35208e50-6550-46e0-8c9d-6e25ddf655ff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.946066 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.945913    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35208e50-6550-46e0-8c9d-6e25ddf655ff-metrics-client-ca\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.946124 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.946096    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/35208e50-6550-46e0-8c9d-6e25ddf655ff-telemeter-client-tls\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.946177 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.946133    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35208e50-6550-46e0-8c9d-6e25ddf655ff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.946289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.946210    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35208e50-6550-46e0-8c9d-6e25ddf655ff-serving-certs-ca-bundle\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.946289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.946252    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/35208e50-6550-46e0-8c9d-6e25ddf655ff-federate-client-tls\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.946289 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.946289    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrxt\" (UniqueName: \"kubernetes.io/projected/35208e50-6550-46e0-8c9d-6e25ddf655ff-kube-api-access-rwrxt\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.946477 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.946335    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/35208e50-6550-46e0-8c9d-6e25ddf655ff-secret-telemeter-client\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.946821 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.946793    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35208e50-6550-46e0-8c9d-6e25ddf655ff-metrics-client-ca\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.947608 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.947561    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35208e50-6550-46e0-8c9d-6e25ddf655ff-serving-certs-ca-bundle\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.947608 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.947577    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35208e50-6550-46e0-8c9d-6e25ddf655ff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.949077 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.949055    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/35208e50-6550-46e0-8c9d-6e25ddf655ff-secret-telemeter-client\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.949318 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.949296    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35208e50-6550-46e0-8c9d-6e25ddf655ff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.949455 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.949425    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/35208e50-6550-46e0-8c9d-6e25ddf655ff-federate-client-tls\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.949573 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.949551    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/35208e50-6550-46e0-8c9d-6e25ddf655ff-telemeter-client-tls\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:36.954861 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:36.954805    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrxt\" (UniqueName: \"kubernetes.io/projected/35208e50-6550-46e0-8c9d-6e25ddf655ff-kube-api-access-rwrxt\") pod \"telemeter-client-5df795b4f-bxd4s\" (UID: \"35208e50-6550-46e0-8c9d-6e25ddf655ff\") " pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:37.020745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:37.020712    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s"
Jul 29 22:51:37.177229 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:37.177139    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5df795b4f-bxd4s"]
Jul 29 22:51:37.179351 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:51:37.179320    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35208e50_6550_46e0_8c9d_6e25ddf655ff.slice/crio-549bef5eb9a8db919bee742a41314668de85b5f3986cddd4905325c1f097b8b4 WatchSource:0}: Error finding container 549bef5eb9a8db919bee742a41314668de85b5f3986cddd4905325c1f097b8b4: Status 404 returned error can't find the container with id 549bef5eb9a8db919bee742a41314668de85b5f3986cddd4905325c1f097b8b4
Jul 29 22:51:37.454259 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:37.454215    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s" event={"ID":"35208e50-6550-46e0-8c9d-6e25ddf655ff","Type":"ContainerStarted","Data":"549bef5eb9a8db919bee742a41314668de85b5f3986cddd4905325c1f097b8b4"}
Jul 29 22:51:38.462232 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:38.462114    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc" event={"ID":"275a9a4a-e06e-40a9-9c9e-45562d22fcbd","Type":"ContainerStarted","Data":"453fd881b9b36469b8e6e02f493b7902671cd8859a8fe104f314f0ece579d9b6"}
Jul 29 22:51:38.479206 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:38.479138    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc" podStartSLOduration=1.765896167 podStartE2EDuration="3.479118054s" totalImagesPullingTime="1.713221887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:38.477589336 +0000 UTC m=+305.482786778" watchObservedRunningTime="2026-07-29 22:51:38.479118054 +0000 UTC m=+305.484315481"
Jul 29 22:51:40.475650 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:40.475608    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s" event={"ID":"35208e50-6550-46e0-8c9d-6e25ddf655ff","Type":"ContainerStarted","Data":"27d3f3687bb56fb97c7580b42afe5eb992ef5ab62f264e34e7bd69728c346031"}
Jul 29 22:51:44.468822 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:44.468780    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:51:45.993434 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:45.993395    2569 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 29 22:51:48.508386 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:48.508351    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-4qmvc" event={"ID":"5e2c4d0c-99fa-45cd-b046-82a40d8a97a0","Type":"ContainerStarted","Data":"92dd3d37bed590387539d6784a801f0433fa1ad6680df5b4ae4caf1f01d3f82a"}
Jul 29 22:51:48.508896 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:48.508531    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-4qmvc"
Jul 29 22:51:48.510247 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:48.510224    2569 patch_prober.go:28] interesting pod/downloads-7c97dc4555-4qmvc container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" start-of-body=
Jul 29 22:51:48.510374 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:48.510269    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-4qmvc" podUID="5e2c4d0c-99fa-45cd-b046-82a40d8a97a0" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused"
Jul 29 22:51:48.510832 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:48.510812    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s" event={"ID":"35208e50-6550-46e0-8c9d-6e25ddf655ff","Type":"ContainerStarted","Data":"141711670f6821f28b82db3a4d541ad55e1dfa23dab876f0540b05c34ea4a2be"}
Jul 29 22:51:48.510910 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:48.510855    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s" event={"ID":"35208e50-6550-46e0-8c9d-6e25ddf655ff","Type":"ContainerStarted","Data":"590f2db989e00138ef114ccec679c51811f61ed9e7287cb5df2eaac53fee0c75"}
Jul 29 22:51:48.524037 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:48.523988    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-4qmvc" podStartSLOduration=0.753632768 podStartE2EDuration="18.523972788s" totalImagesPullingTime="17.77034002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:48.523777871 +0000 UTC m=+315.528975321" watchObservedRunningTime="2026-07-29 22:51:48.523972788 +0000 UTC m=+315.529170210"
Jul 29 22:51:48.544543 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:48.544427    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5df795b4f-bxd4s" podStartSLOduration=1.5005743969999998 podStartE2EDuration="12.544404895s" totalImagesPullingTime="11.043830498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:51:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:48.542656529 +0000 UTC m=+315.547853951" watchObservedRunningTime="2026-07-29 22:51:48.544404895 +0000 UTC m=+315.549602320"
Jul 29 22:51:49.535726 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:49.535693    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-4qmvc"
Jul 29 22:51:55.763576 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:55.763542    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-lzctb_8088e072-67e5-4ff4-a931-e881cb0e69fa/kube-state-metrics/0.log"
Jul 29 22:51:55.963713 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:55.963683    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-lzctb_8088e072-67e5-4ff4-a931-e881cb0e69fa/kube-rbac-proxy-main/0.log"
Jul 29 22:51:55.992762 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:55.992713    2569 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 29 22:51:55.992938 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:55.992792    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 22:51:55.993439 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:55.993396    2569 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 29 22:51:55.993553 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:55.993468    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="service-proxy" containerID="cri-o://73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd" gracePeriod=30
Jul 29 22:51:56.163713 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:56.163688    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-lzctb_8088e072-67e5-4ff4-a931-e881cb0e69fa/kube-rbac-proxy-self/0.log"
Jul 29 22:51:56.242008 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:56.241970    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:56.242167 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:56.242043    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:51:56.363067 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:56.362980    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bb87447bb-z4kjc_275a9a4a-e06e-40a9-9c9e-45562d22fcbd/metrics-server/0.log"
Jul 29 22:51:56.551503 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:56.551442    2569 generic.go:356] "Generic (PLEG): container finished" podID="968c59fb-1819-4280-8671-cd892f0d060a" containerID="73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd" exitCode=2
Jul 29 22:51:56.551684 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:56.551525    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" event={"ID":"968c59fb-1819-4280-8671-cd892f0d060a","Type":"ContainerDied","Data":"73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd"}
Jul 29 22:51:56.551684 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:56.551578    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" event={"ID":"968c59fb-1819-4280-8671-cd892f0d060a","Type":"ContainerStarted","Data":"62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7"}
Jul 29 22:51:57.963071 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:57.963043    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kn7nm_704b9290-51eb-4000-9ab2-7ae30d6f8764/init-textfile/0.log"
Jul 29 22:51:58.163909 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:58.163879    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kn7nm_704b9290-51eb-4000-9ab2-7ae30d6f8764/node-exporter/0.log"
Jul 29 22:51:58.363589 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:51:58.363554    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kn7nm_704b9290-51eb-4000-9ab2-7ae30d6f8764/kube-rbac-proxy/0.log"
Jul 29 22:52:01.162798 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:01.162753    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5df795b4f-bxd4s_35208e50-6550-46e0-8c9d-6e25ddf655ff/telemeter-client/0.log"
Jul 29 22:52:01.363045 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:01.363012    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5df795b4f-bxd4s_35208e50-6550-46e0-8c9d-6e25ddf655ff/reload/0.log"
Jul 29 22:52:01.564622 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:01.564591    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5df795b4f-bxd4s_35208e50-6550-46e0-8c9d-6e25ddf655ff/kube-rbac-proxy/0.log"
Jul 29 22:52:03.162610 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:03.162575    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 22:52:03.366780 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:03.366745    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/3.log"
Jul 29 22:52:04.165046 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:04.165021    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-4qmvc_5e2c4d0c-99fa-45cd-b046-82a40d8a97a0/download-server/0.log"
Jul 29 22:52:16.246904 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:16.246876    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:52:16.250641 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:16.250617    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7bb87447bb-z4kjc"
Jul 29 22:52:47.253692 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:47.253630    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:52:47.255937 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:47.255910    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d19dc00-e3da-4306-a5ff-8cca92e02a1c-metrics-certs\") pod \"network-metrics-daemon-t5x5r\" (UID: \"0d19dc00-e3da-4306-a5ff-8cca92e02a1c\") " pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:52:47.471081 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:47.471048    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wkbjs\""
Jul 29 22:52:47.479315 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:47.479289    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5x5r"
Jul 29 22:52:47.606263 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:47.606238    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t5x5r"]
Jul 29 22:52:47.608911 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:52:47.608887    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d19dc00_e3da_4306_a5ff_8cca92e02a1c.slice/crio-d14ecf63286d520de655b62ac1881bc295190e59312fa7ea2fd7a49dfbfa53ce WatchSource:0}: Error finding container d14ecf63286d520de655b62ac1881bc295190e59312fa7ea2fd7a49dfbfa53ce: Status 404 returned error can't find the container with id d14ecf63286d520de655b62ac1881bc295190e59312fa7ea2fd7a49dfbfa53ce
Jul 29 22:52:47.730082 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:47.730046    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t5x5r" event={"ID":"0d19dc00-e3da-4306-a5ff-8cca92e02a1c","Type":"ContainerStarted","Data":"d14ecf63286d520de655b62ac1881bc295190e59312fa7ea2fd7a49dfbfa53ce"}
Jul 29 22:52:48.735532 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:48.735475    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t5x5r" event={"ID":"0d19dc00-e3da-4306-a5ff-8cca92e02a1c","Type":"ContainerStarted","Data":"6b448bc7c73a57c3f69a53072a27fe16efd8f0980b8a7f3b9829a02e93c08ddf"}
Jul 29 22:52:48.735917 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:48.735551    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t5x5r" event={"ID":"0d19dc00-e3da-4306-a5ff-8cca92e02a1c","Type":"ContainerStarted","Data":"b702157dc875bfe93c34984b5d1b4f62f0dd5c39c043c9e45192f50dfd7f0abb"}
Jul 29 22:52:48.750577 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:52:48.750522    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t5x5r" podStartSLOduration=374.907035842 podStartE2EDuration="6m15.750504724s" totalImagesPullingTime="843.468882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:52:48.749447998 +0000 UTC m=+375.754645422" watchObservedRunningTime="2026-07-29 22:52:48.750504724 +0000 UTC m=+375.755702147"
Jul 29 22:54:45.385339 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.385306    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"]
Jul 29 22:54:45.388534 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.388512    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"
Jul 29 22:54:45.391137 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.391116    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 29 22:54:45.391252 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.391177    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-74tcb\""
Jul 29 22:54:45.391699 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.391682    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 29 22:54:45.391848 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.391830    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 29 22:54:45.392338 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.392321    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 29 22:54:45.392395 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.392329    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 29 22:54:45.399959 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.399928    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"]
Jul 29 22:54:45.452003 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.451972    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3ceb3c1b-bc88-41bb-9dca-d0704f0bd258-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tnf9z\" (UID: \"3ceb3c1b-bc88-41bb-9dca-d0704f0bd258\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"
Jul 29 22:54:45.452157 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.452039    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ceb3c1b-bc88-41bb-9dca-d0704f0bd258-certificates\") pod \"keda-metrics-apiserver-55557df8c-tnf9z\" (UID: \"3ceb3c1b-bc88-41bb-9dca-d0704f0bd258\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"
Jul 29 22:54:45.452157 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.452063    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wdnn\" (UniqueName: \"kubernetes.io/projected/3ceb3c1b-bc88-41bb-9dca-d0704f0bd258-kube-api-access-4wdnn\") pod \"keda-metrics-apiserver-55557df8c-tnf9z\" (UID: \"3ceb3c1b-bc88-41bb-9dca-d0704f0bd258\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"
Jul 29 22:54:45.553337 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.553301    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ceb3c1b-bc88-41bb-9dca-d0704f0bd258-certificates\") pod \"keda-metrics-apiserver-55557df8c-tnf9z\" (UID: \"3ceb3c1b-bc88-41bb-9dca-d0704f0bd258\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"
Jul 29 22:54:45.553337 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.553336    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wdnn\" (UniqueName: \"kubernetes.io/projected/3ceb3c1b-bc88-41bb-9dca-d0704f0bd258-kube-api-access-4wdnn\") pod \"keda-metrics-apiserver-55557df8c-tnf9z\" (UID: \"3ceb3c1b-bc88-41bb-9dca-d0704f0bd258\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"
Jul 29 22:54:45.553565 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.553382    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3ceb3c1b-bc88-41bb-9dca-d0704f0bd258-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tnf9z\" (UID: \"3ceb3c1b-bc88-41bb-9dca-d0704f0bd258\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"
Jul 29 22:54:45.553565 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:54:45.553438    2569 secret.go:281] references non-existent secret key: tls.crt
Jul 29 22:54:45.553565 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:54:45.553456    2569 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 29 22:54:45.553565 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:54:45.553471    2569 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 29 22:54:45.553565 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:54:45.553504    2569 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 29 22:54:45.553565 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:54:45.553557    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ceb3c1b-bc88-41bb-9dca-d0704f0bd258-certificates podName:3ceb3c1b-bc88-41bb-9dca-d0704f0bd258 nodeName:}" failed. No retries permitted until 2026-07-29 22:54:46.053541824 +0000 UTC m=+493.058739224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3ceb3c1b-bc88-41bb-9dca-d0704f0bd258-certificates") pod "keda-metrics-apiserver-55557df8c-tnf9z" (UID: "3ceb3c1b-bc88-41bb-9dca-d0704f0bd258") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 29 22:54:45.553806 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.553778    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3ceb3c1b-bc88-41bb-9dca-d0704f0bd258-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tnf9z\" (UID: \"3ceb3c1b-bc88-41bb-9dca-d0704f0bd258\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"
Jul 29 22:54:45.563977 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:45.563957    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wdnn\" (UniqueName: \"kubernetes.io/projected/3ceb3c1b-bc88-41bb-9dca-d0704f0bd258-kube-api-access-4wdnn\") pod \"keda-metrics-apiserver-55557df8c-tnf9z\" (UID: \"3ceb3c1b-bc88-41bb-9dca-d0704f0bd258\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"
Jul 29 22:54:46.058046 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:46.058015    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ceb3c1b-bc88-41bb-9dca-d0704f0bd258-certificates\") pod \"keda-metrics-apiserver-55557df8c-tnf9z\" (UID: \"3ceb3c1b-bc88-41bb-9dca-d0704f0bd258\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"
Jul 29 22:54:46.060468 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:46.060445    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ceb3c1b-bc88-41bb-9dca-d0704f0bd258-certificates\") pod \"keda-metrics-apiserver-55557df8c-tnf9z\" (UID: \"3ceb3c1b-bc88-41bb-9dca-d0704f0bd258\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"
Jul 29 22:54:46.298254 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:46.298217    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"
Jul 29 22:54:46.432756 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:46.432721    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"]
Jul 29 22:54:46.436266 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:54:46.436241    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ceb3c1b_bc88_41bb_9dca_d0704f0bd258.slice/crio-3af31e10d0ce7a8cdd2c6f83afad93e6b18b78fceea7b0fb61d634e2df7bd551 WatchSource:0}: Error finding container 3af31e10d0ce7a8cdd2c6f83afad93e6b18b78fceea7b0fb61d634e2df7bd551: Status 404 returned error can't find the container with id 3af31e10d0ce7a8cdd2c6f83afad93e6b18b78fceea7b0fb61d634e2df7bd551
Jul 29 22:54:47.153140 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:47.153101    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z" event={"ID":"3ceb3c1b-bc88-41bb-9dca-d0704f0bd258","Type":"ContainerStarted","Data":"3af31e10d0ce7a8cdd2c6f83afad93e6b18b78fceea7b0fb61d634e2df7bd551"}
Jul 29 22:54:50.165873 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:50.165834    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z" event={"ID":"3ceb3c1b-bc88-41bb-9dca-d0704f0bd258","Type":"ContainerStarted","Data":"6d86b0335b650e68be5fcb3441bde88905f8c567b879e52d888ed66ad88bfeec"}
Jul 29 22:54:50.166278 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:50.166089    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"
Jul 29 22:54:50.182253 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:54:50.182204    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z" podStartSLOduration=2.3011609330000002 podStartE2EDuration="5.182189223s" totalImagesPullingTime="2.88102829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:54:50.181030952 +0000 UTC m=+497.186228377" watchObservedRunningTime="2026-07-29 22:54:50.182189223 +0000 UTC m=+497.187386645"
Jul 29 22:55:01.173686 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:55:01.173652    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tnf9z"
Jul 29 22:56:03.878896 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:03.878863    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"]
Jul 29 22:56:03.881924 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:03.881906    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"
Jul 29 22:56:03.883898 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:03.883878    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 29 22:56:03.884013 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:03.883979    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 29 22:56:03.884582 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:03.884560    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-hhvxm\""
Jul 29 22:56:03.884582 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:03.884574    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 29 22:56:03.892212 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:03.892189    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"]
Jul 29 22:56:03.982330 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:03.982294    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f759b397-beac-4707-ba7f-1c1bb88128c1-cert\") pod \"llmisvc-controller-manager-68fcfb49c7-g5w7l\" (UID: \"f759b397-beac-4707-ba7f-1c1bb88128c1\") " pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"
Jul 29 22:56:03.982528 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:03.982339    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sqg9\" (UniqueName: \"kubernetes.io/projected/f759b397-beac-4707-ba7f-1c1bb88128c1-kube-api-access-5sqg9\") pod \"llmisvc-controller-manager-68fcfb49c7-g5w7l\" (UID: \"f759b397-beac-4707-ba7f-1c1bb88128c1\") " pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"
Jul 29 22:56:04.083726 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:04.083690    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f759b397-beac-4707-ba7f-1c1bb88128c1-cert\") pod \"llmisvc-controller-manager-68fcfb49c7-g5w7l\" (UID: \"f759b397-beac-4707-ba7f-1c1bb88128c1\") " pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"
Jul 29 22:56:04.083886 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:04.083737    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sqg9\" (UniqueName: \"kubernetes.io/projected/f759b397-beac-4707-ba7f-1c1bb88128c1-kube-api-access-5sqg9\") pod \"llmisvc-controller-manager-68fcfb49c7-g5w7l\" (UID: \"f759b397-beac-4707-ba7f-1c1bb88128c1\") " pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"
Jul 29 22:56:04.086139 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:04.086111    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f759b397-beac-4707-ba7f-1c1bb88128c1-cert\") pod \"llmisvc-controller-manager-68fcfb49c7-g5w7l\" (UID: \"f759b397-beac-4707-ba7f-1c1bb88128c1\") " pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"
Jul 29 22:56:04.095527 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:04.095502    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sqg9\" (UniqueName: \"kubernetes.io/projected/f759b397-beac-4707-ba7f-1c1bb88128c1-kube-api-access-5sqg9\") pod \"llmisvc-controller-manager-68fcfb49c7-g5w7l\" (UID: \"f759b397-beac-4707-ba7f-1c1bb88128c1\") " pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"
Jul 29 22:56:04.192722 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:04.192635    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"
Jul 29 22:56:04.318354 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:04.318330    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"]
Jul 29 22:56:04.321010 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:56:04.320980    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf759b397_beac_4707_ba7f_1c1bb88128c1.slice/crio-80385aed1935413d7012f0421e6efa4ed94f2c0e97da3ce5cc1135dabd393f54 WatchSource:0}: Error finding container 80385aed1935413d7012f0421e6efa4ed94f2c0e97da3ce5cc1135dabd393f54: Status 404 returned error can't find the container with id 80385aed1935413d7012f0421e6efa4ed94f2c0e97da3ce5cc1135dabd393f54
Jul 29 22:56:04.426635 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:04.426596    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l" event={"ID":"f759b397-beac-4707-ba7f-1c1bb88128c1","Type":"ContainerStarted","Data":"80385aed1935413d7012f0421e6efa4ed94f2c0e97da3ce5cc1135dabd393f54"}
Jul 29 22:56:07.440376 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:07.440342    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l" event={"ID":"f759b397-beac-4707-ba7f-1c1bb88128c1","Type":"ContainerStarted","Data":"0f46d27ecbfd6ca896fbb51eb6138a777ad94462147567115085e808c20a3356"}
Jul 29 22:56:07.440756 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:07.440580    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"
Jul 29 22:56:07.456882 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:07.456835    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l" podStartSLOduration=1.548096803 podStartE2EDuration="4.456818759s" totalImagesPullingTime="2.908721956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:56:07.455527882 +0000 UTC m=+574.460725308" watchObservedRunningTime="2026-07-29 22:56:07.456818759 +0000 UTC m=+574.462016182"
Jul 29 22:56:33.437826 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:33.437796    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 22:56:33.438266 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:33.437796    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 22:56:38.446697 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:56:38.446667    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"
Jul 29 22:57:12.505312 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:12.505235    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nwwdq"]
Jul 29 22:57:12.508356 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:12.508334    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nwwdq"
Jul 29 22:57:12.510466 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:12.510444    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-86s5b\""
Jul 29 22:57:12.510589 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:12.510559    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 29 22:57:12.510901 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:12.510886    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 29 22:57:12.519039 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:12.519015    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nwwdq"]
Jul 29 22:57:12.636391 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:12.636346    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a105be3d-bac5-42b3-afff-7926d7c97688-bound-sa-token\") pod \"cert-manager-55b4d954d5-nwwdq\" (UID: \"a105be3d-bac5-42b3-afff-7926d7c97688\") " pod="cert-manager/cert-manager-55b4d954d5-nwwdq"
Jul 29 22:57:12.636391 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:12.636390    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hdvt\" (UniqueName: \"kubernetes.io/projected/a105be3d-bac5-42b3-afff-7926d7c97688-kube-api-access-6hdvt\") pod \"cert-manager-55b4d954d5-nwwdq\" (UID: \"a105be3d-bac5-42b3-afff-7926d7c97688\") " pod="cert-manager/cert-manager-55b4d954d5-nwwdq"
Jul 29 22:57:12.737027 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:12.736991    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a105be3d-bac5-42b3-afff-7926d7c97688-bound-sa-token\") pod \"cert-manager-55b4d954d5-nwwdq\" (UID: \"a105be3d-bac5-42b3-afff-7926d7c97688\") " pod="cert-manager/cert-manager-55b4d954d5-nwwdq"
Jul 29 22:57:12.737027 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:12.737028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hdvt\" (UniqueName: \"kubernetes.io/projected/a105be3d-bac5-42b3-afff-7926d7c97688-kube-api-access-6hdvt\") pod \"cert-manager-55b4d954d5-nwwdq\" (UID: \"a105be3d-bac5-42b3-afff-7926d7c97688\") " pod="cert-manager/cert-manager-55b4d954d5-nwwdq"
Jul 29 22:57:12.744542 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:12.744507    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a105be3d-bac5-42b3-afff-7926d7c97688-bound-sa-token\") pod \"cert-manager-55b4d954d5-nwwdq\" (UID: \"a105be3d-bac5-42b3-afff-7926d7c97688\") " pod="cert-manager/cert-manager-55b4d954d5-nwwdq"
Jul 29 22:57:12.744799 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:12.744779    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hdvt\" (UniqueName: \"kubernetes.io/projected/a105be3d-bac5-42b3-afff-7926d7c97688-kube-api-access-6hdvt\") pod \"cert-manager-55b4d954d5-nwwdq\" (UID: \"a105be3d-bac5-42b3-afff-7926d7c97688\") " pod="cert-manager/cert-manager-55b4d954d5-nwwdq"
Jul 29 22:57:12.817420 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:12.817340    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nwwdq"
Jul 29 22:57:12.940516 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:12.940473    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nwwdq"]
Jul 29 22:57:12.943541 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:57:12.943507    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda105be3d_bac5_42b3_afff_7926d7c97688.slice/crio-52d460544d4eaee53579719ea23501d1f8063d5502a07a6c3ac974cbe8c5287e WatchSource:0}: Error finding container 52d460544d4eaee53579719ea23501d1f8063d5502a07a6c3ac974cbe8c5287e: Status 404 returned error can't find the container with id 52d460544d4eaee53579719ea23501d1f8063d5502a07a6c3ac974cbe8c5287e
Jul 29 22:57:12.945370 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:12.945356    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 22:57:13.669199 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:13.669161    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nwwdq" event={"ID":"a105be3d-bac5-42b3-afff-7926d7c97688","Type":"ContainerStarted","Data":"52d460544d4eaee53579719ea23501d1f8063d5502a07a6c3ac974cbe8c5287e"}
Jul 29 22:57:16.682823 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:16.682789    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nwwdq" event={"ID":"a105be3d-bac5-42b3-afff-7926d7c97688","Type":"ContainerStarted","Data":"c7161ca39c9f38be9a96213d7de725c4cfbbde5750c86e532509de9d74f89b03"}
Jul 29 22:57:16.704155 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:16.704106    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-nwwdq" podStartSLOduration=1.1121298450000001 podStartE2EDuration="4.704089058s" totalImagesPullingTime="3.591959213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:57:16.702080933 +0000 UTC m=+643.707278367" watchObservedRunningTime="2026-07-29 22:57:16.704089058 +0000 UTC m=+643.709286481"
Jul 29 22:57:36.921309 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:36.921278    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"]
Jul 29 22:57:36.924409 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:36.924390    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:57:36.927315 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:36.927283    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 22:57:36.927458 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:36.927331    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 22:57:36.927579 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:36.927565    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 22:57:36.927639 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:36.927609    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 22:57:36.927822 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:36.927808    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-v66rt\""
Jul 29 22:57:36.927899 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:36.927869    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 22:57:36.934868 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:36.934845    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"]
Jul 29 22:57:37.034554 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:37.034510    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4fc1b98b-b6e0-482b-afad-dd4c0db5d603-manager-config\") pod \"lws-controller-manager-78dcd69c74-5zff2\" (UID: \"4fc1b98b-b6e0-482b-afad-dd4c0db5d603\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:57:37.034554 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:37.034552    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fc1b98b-b6e0-482b-afad-dd4c0db5d603-cert\") pod \"lws-controller-manager-78dcd69c74-5zff2\" (UID: \"4fc1b98b-b6e0-482b-afad-dd4c0db5d603\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:57:37.034755 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:37.034622    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpjnj\" (UniqueName: \"kubernetes.io/projected/4fc1b98b-b6e0-482b-afad-dd4c0db5d603-kube-api-access-lpjnj\") pod \"lws-controller-manager-78dcd69c74-5zff2\" (UID: \"4fc1b98b-b6e0-482b-afad-dd4c0db5d603\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:57:37.034755 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:37.034698    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc1b98b-b6e0-482b-afad-dd4c0db5d603-metrics-cert\") pod \"lws-controller-manager-78dcd69c74-5zff2\" (UID: \"4fc1b98b-b6e0-482b-afad-dd4c0db5d603\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:57:37.135634 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:37.135598    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4fc1b98b-b6e0-482b-afad-dd4c0db5d603-manager-config\") pod \"lws-controller-manager-78dcd69c74-5zff2\" (UID: \"4fc1b98b-b6e0-482b-afad-dd4c0db5d603\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:57:37.135763 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:37.135668    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fc1b98b-b6e0-482b-afad-dd4c0db5d603-cert\") pod \"lws-controller-manager-78dcd69c74-5zff2\" (UID: \"4fc1b98b-b6e0-482b-afad-dd4c0db5d603\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:57:37.135763 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:37.135707    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpjnj\" (UniqueName: \"kubernetes.io/projected/4fc1b98b-b6e0-482b-afad-dd4c0db5d603-kube-api-access-lpjnj\") pod \"lws-controller-manager-78dcd69c74-5zff2\" (UID: \"4fc1b98b-b6e0-482b-afad-dd4c0db5d603\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:57:37.135763 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:37.135753    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc1b98b-b6e0-482b-afad-dd4c0db5d603-metrics-cert\") pod \"lws-controller-manager-78dcd69c74-5zff2\" (UID: \"4fc1b98b-b6e0-482b-afad-dd4c0db5d603\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:57:37.136090 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:37.136067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4fc1b98b-b6e0-482b-afad-dd4c0db5d603-manager-config\") pod \"lws-controller-manager-78dcd69c74-5zff2\" (UID: \"4fc1b98b-b6e0-482b-afad-dd4c0db5d603\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:57:37.138093 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:37.138068    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc1b98b-b6e0-482b-afad-dd4c0db5d603-metrics-cert\") pod \"lws-controller-manager-78dcd69c74-5zff2\" (UID: \"4fc1b98b-b6e0-482b-afad-dd4c0db5d603\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:57:37.138195 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:37.138142    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fc1b98b-b6e0-482b-afad-dd4c0db5d603-cert\") pod \"lws-controller-manager-78dcd69c74-5zff2\" (UID: \"4fc1b98b-b6e0-482b-afad-dd4c0db5d603\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:57:37.146705 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:37.146675    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpjnj\" (UniqueName: \"kubernetes.io/projected/4fc1b98b-b6e0-482b-afad-dd4c0db5d603-kube-api-access-lpjnj\") pod \"lws-controller-manager-78dcd69c74-5zff2\" (UID: \"4fc1b98b-b6e0-482b-afad-dd4c0db5d603\") " pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:57:37.234554 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:37.234518    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:57:37.376552 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:37.376492    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"]
Jul 29 22:57:37.378964 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:57:37.378929    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc1b98b_b6e0_482b_afad_dd4c0db5d603.slice/crio-98b5e88dcc477a67d052a257ac7413a34747163a71ee7100c25a046bdcac21a6 WatchSource:0}: Error finding container 98b5e88dcc477a67d052a257ac7413a34747163a71ee7100c25a046bdcac21a6: Status 404 returned error can't find the container with id 98b5e88dcc477a67d052a257ac7413a34747163a71ee7100c25a046bdcac21a6
Jul 29 22:57:37.759957 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:37.759921    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2" event={"ID":"4fc1b98b-b6e0-482b-afad-dd4c0db5d603","Type":"ContainerStarted","Data":"98b5e88dcc477a67d052a257ac7413a34747163a71ee7100c25a046bdcac21a6"}
Jul 29 22:57:40.774762 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:40.774728    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2" event={"ID":"4fc1b98b-b6e0-482b-afad-dd4c0db5d603","Type":"ContainerStarted","Data":"8572e9124457042ddde9095f55f024936a4111ca61bbf89182927988cb319464"}
Jul 29 22:57:40.775159 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:40.774829    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:57:40.791807 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:40.791732    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2" podStartSLOduration=2.237588481 podStartE2EDuration="4.791717351s" totalImagesPullingTime="2.55412887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:57:40.790575418 +0000 UTC m=+667.795772855" watchObservedRunningTime="2026-07-29 22:57:40.791717351 +0000 UTC m=+667.796914773"
Jul 29 22:57:51.781107 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:57:51.781074    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-78dcd69c74-5zff2"
Jul 29 22:58:25.771395 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.771318    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns"]
Jul 29 22:58:25.777741 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.777719    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns"
Jul 29 22:58:25.780389 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.780363    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 29 22:58:25.780389 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.780380    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 22:58:25.780578 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.780458    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 29 22:58:25.781255 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.781238    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m864v\""
Jul 29 22:58:25.781297 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.781275    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 22:58:25.793031 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.793005    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns"]
Jul 29 22:58:25.859822 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.859788    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd854\" (UniqueName: \"kubernetes.io/projected/7a212016-b3bc-4ad1-8b8d-ce9134050cfe-kube-api-access-vd854\") pod \"kuadrant-console-plugin-598f86468b-vrnns\" (UID: \"7a212016-b3bc-4ad1-8b8d-ce9134050cfe\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns"
Jul 29 22:58:25.860018 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.859832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a212016-b3bc-4ad1-8b8d-ce9134050cfe-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-vrnns\" (UID: \"7a212016-b3bc-4ad1-8b8d-ce9134050cfe\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns"
Jul 29 22:58:25.860018 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.859881    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a212016-b3bc-4ad1-8b8d-ce9134050cfe-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-vrnns\" (UID: \"7a212016-b3bc-4ad1-8b8d-ce9134050cfe\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns"
Jul 29 22:58:25.960742 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.960713    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a212016-b3bc-4ad1-8b8d-ce9134050cfe-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-vrnns\" (UID: \"7a212016-b3bc-4ad1-8b8d-ce9134050cfe\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns"
Jul 29 22:58:25.960892 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.960758    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a212016-b3bc-4ad1-8b8d-ce9134050cfe-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-vrnns\" (UID: \"7a212016-b3bc-4ad1-8b8d-ce9134050cfe\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns"
Jul 29 22:58:25.960892 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.960813    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd854\" (UniqueName: \"kubernetes.io/projected/7a212016-b3bc-4ad1-8b8d-ce9134050cfe-kube-api-access-vd854\") pod \"kuadrant-console-plugin-598f86468b-vrnns\" (UID: \"7a212016-b3bc-4ad1-8b8d-ce9134050cfe\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns"
Jul 29 22:58:25.960979 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:58:25.960922    2569 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 29 22:58:25.961017 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:58:25.960985    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a212016-b3bc-4ad1-8b8d-ce9134050cfe-plugin-serving-cert podName:7a212016-b3bc-4ad1-8b8d-ce9134050cfe nodeName:}" failed. No retries permitted until 2026-07-29 22:58:26.460963055 +0000 UTC m=+713.466160471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/7a212016-b3bc-4ad1-8b8d-ce9134050cfe-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-vrnns" (UID: "7a212016-b3bc-4ad1-8b8d-ce9134050cfe") : secret "plugin-serving-cert" not found
Jul 29 22:58:25.961381 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.961362    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a212016-b3bc-4ad1-8b8d-ce9134050cfe-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-vrnns\" (UID: \"7a212016-b3bc-4ad1-8b8d-ce9134050cfe\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns"
Jul 29 22:58:25.969568 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:25.969540    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd854\" (UniqueName: \"kubernetes.io/projected/7a212016-b3bc-4ad1-8b8d-ce9134050cfe-kube-api-access-vd854\") pod \"kuadrant-console-plugin-598f86468b-vrnns\" (UID: \"7a212016-b3bc-4ad1-8b8d-ce9134050cfe\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns"
Jul 29 22:58:26.465311 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:26.465273    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a212016-b3bc-4ad1-8b8d-ce9134050cfe-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-vrnns\" (UID: \"7a212016-b3bc-4ad1-8b8d-ce9134050cfe\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns"
Jul 29 22:58:26.467723 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:26.467702    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a212016-b3bc-4ad1-8b8d-ce9134050cfe-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-vrnns\" (UID: \"7a212016-b3bc-4ad1-8b8d-ce9134050cfe\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns"
Jul 29 22:58:26.686851 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:26.686812    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns"
Jul 29 22:58:26.813813 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:26.813781    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns"]
Jul 29 22:58:26.817893 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:58:26.817863    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a212016_b3bc_4ad1_8b8d_ce9134050cfe.slice/crio-56cf59b657ceae4738132bd5ebe185dfd567602a01c4b0f4a665ea82024c1402 WatchSource:0}: Error finding container 56cf59b657ceae4738132bd5ebe185dfd567602a01c4b0f4a665ea82024c1402: Status 404 returned error can't find the container with id 56cf59b657ceae4738132bd5ebe185dfd567602a01c4b0f4a665ea82024c1402
Jul 29 22:58:26.948281 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:26.948245    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns" event={"ID":"7a212016-b3bc-4ad1-8b8d-ce9134050cfe","Type":"ContainerStarted","Data":"56cf59b657ceae4738132bd5ebe185dfd567602a01c4b0f4a665ea82024c1402"}
Jul 29 22:58:34.989713 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:34.989674    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns" event={"ID":"7a212016-b3bc-4ad1-8b8d-ce9134050cfe","Type":"ContainerStarted","Data":"107dc4dcab04bec9d163594f6b1fa1c71d8fd91b9dc8c40d9ac8c99a47a7d9aa"}
Jul 29 22:58:35.005431 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:58:35.005366    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vrnns" podStartSLOduration=2.778120231 podStartE2EDuration="10.005351119s" totalImagesPullingTime="7.227230888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:58:35.004374394 +0000 UTC m=+722.009571834" watchObservedRunningTime="2026-07-29 22:58:35.005351119 +0000 UTC m=+722.010548545"
Jul 29 22:59:11.121728 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.121694    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-c4njl"]
Jul 29 22:59:11.146883 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.146838    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-c4njl"]
Jul 29 22:59:11.147051 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.146936    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-c4njl"
Jul 29 22:59:11.149271 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.149249    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8w8rg\""
Jul 29 22:59:11.235533 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.235470    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh46v\" (UniqueName: \"kubernetes.io/projected/2deaa695-e9d6-4f83-b733-4125debcd73e-kube-api-access-mh46v\") pod \"authorino-b54df94b4-c4njl\" (UID: \"2deaa695-e9d6-4f83-b733-4125debcd73e\") " pod="kuadrant-system/authorino-b54df94b4-c4njl"
Jul 29 22:59:11.250264 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.250231    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-xvmlj"]
Jul 29 22:59:11.253579 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.253557    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-xvmlj"
Jul 29 22:59:11.259409 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.259359    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-xvmlj"]
Jul 29 22:59:11.335993 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.335950    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh46v\" (UniqueName: \"kubernetes.io/projected/2deaa695-e9d6-4f83-b733-4125debcd73e-kube-api-access-mh46v\") pod \"authorino-b54df94b4-c4njl\" (UID: \"2deaa695-e9d6-4f83-b733-4125debcd73e\") " pod="kuadrant-system/authorino-b54df94b4-c4njl"
Jul 29 22:59:11.336191 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.336040    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khnhg\" (UniqueName: \"kubernetes.io/projected/c601d8ad-d8f6-4a5c-8508-daea32ddd5b8-kube-api-access-khnhg\") pod \"authorino-c548b5d7d-xvmlj\" (UID: \"c601d8ad-d8f6-4a5c-8508-daea32ddd5b8\") " pod="kuadrant-system/authorino-c548b5d7d-xvmlj"
Jul 29 22:59:11.343601 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.343569    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh46v\" (UniqueName: \"kubernetes.io/projected/2deaa695-e9d6-4f83-b733-4125debcd73e-kube-api-access-mh46v\") pod \"authorino-b54df94b4-c4njl\" (UID: \"2deaa695-e9d6-4f83-b733-4125debcd73e\") " pod="kuadrant-system/authorino-b54df94b4-c4njl"
Jul 29 22:59:11.437074 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.436989    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khnhg\" (UniqueName: \"kubernetes.io/projected/c601d8ad-d8f6-4a5c-8508-daea32ddd5b8-kube-api-access-khnhg\") pod \"authorino-c548b5d7d-xvmlj\" (UID: \"c601d8ad-d8f6-4a5c-8508-daea32ddd5b8\") " pod="kuadrant-system/authorino-c548b5d7d-xvmlj"
Jul 29 22:59:11.445345 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.445321    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khnhg\" (UniqueName: \"kubernetes.io/projected/c601d8ad-d8f6-4a5c-8508-daea32ddd5b8-kube-api-access-khnhg\") pod \"authorino-c548b5d7d-xvmlj\" (UID: \"c601d8ad-d8f6-4a5c-8508-daea32ddd5b8\") " pod="kuadrant-system/authorino-c548b5d7d-xvmlj"
Jul 29 22:59:11.455115 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.455088    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-c4njl"
Jul 29 22:59:11.562809 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.562779    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-xvmlj"
Jul 29 22:59:11.591834 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.591800    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-c4njl"]
Jul 29 22:59:11.596330 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:59:11.596294    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2deaa695_e9d6_4f83_b733_4125debcd73e.slice/crio-843f33a122c149464c35a170189442db76bf9f62b910b81c70ee147479e6ba69 WatchSource:0}: Error finding container 843f33a122c149464c35a170189442db76bf9f62b910b81c70ee147479e6ba69: Status 404 returned error can't find the container with id 843f33a122c149464c35a170189442db76bf9f62b910b81c70ee147479e6ba69
Jul 29 22:59:11.695737 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:11.695515    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-xvmlj"]
Jul 29 22:59:11.698225 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:59:11.698191    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc601d8ad_d8f6_4a5c_8508_daea32ddd5b8.slice/crio-628c64ac2e43273d04aa2128b0e06213ff856d3094d000241f3adc0de2255fb7 WatchSource:0}: Error finding container 628c64ac2e43273d04aa2128b0e06213ff856d3094d000241f3adc0de2255fb7: Status 404 returned error can't find the container with id 628c64ac2e43273d04aa2128b0e06213ff856d3094d000241f3adc0de2255fb7
Jul 29 22:59:12.129641 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:12.129604    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-xvmlj" event={"ID":"c601d8ad-d8f6-4a5c-8508-daea32ddd5b8","Type":"ContainerStarted","Data":"628c64ac2e43273d04aa2128b0e06213ff856d3094d000241f3adc0de2255fb7"}
Jul 29 22:59:12.130745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:12.130720    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-c4njl" event={"ID":"2deaa695-e9d6-4f83-b733-4125debcd73e","Type":"ContainerStarted","Data":"843f33a122c149464c35a170189442db76bf9f62b910b81c70ee147479e6ba69"}
Jul 29 22:59:15.147810 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:15.147774    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-c4njl" event={"ID":"2deaa695-e9d6-4f83-b733-4125debcd73e","Type":"ContainerStarted","Data":"23598ad4a3736c276a6930fc331556adb2e8e9076b27871ca5110ee737030e2b"}
Jul 29 22:59:15.149220 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:15.149188    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-xvmlj" event={"ID":"c601d8ad-d8f6-4a5c-8508-daea32ddd5b8","Type":"ContainerStarted","Data":"9747ae3a1c931f3dfc9912b4db1896392bc39417429baeabf3f849b9a966ab98"}
Jul 29 22:59:15.161366 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:15.161322    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-c4njl" podStartSLOduration=1.451462934 podStartE2EDuration="4.16130453s" totalImagesPullingTime="2.709841596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:59:15.160135172 +0000 UTC m=+762.165332596" watchObservedRunningTime="2026-07-29 22:59:15.16130453 +0000 UTC m=+762.166501953"
Jul 29 22:59:15.173177 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:15.173112    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-xvmlj" podStartSLOduration=1.554748392 podStartE2EDuration="4.173091221s" totalImagesPullingTime="2.618342829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:59:15.1724582 +0000 UTC m=+762.177655624" watchObservedRunningTime="2026-07-29 22:59:15.173091221 +0000 UTC m=+762.178288648"
Jul 29 22:59:15.194633 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:15.194595    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-c4njl"]
Jul 29 22:59:17.157066 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:17.157022    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-c4njl" podUID="2deaa695-e9d6-4f83-b733-4125debcd73e" containerName="authorino" containerID="cri-o://23598ad4a3736c276a6930fc331556adb2e8e9076b27871ca5110ee737030e2b" gracePeriod=30
Jul 29 22:59:17.415044 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:17.414979    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-c4njl"
Jul 29 22:59:17.491154 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:17.491124    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh46v\" (UniqueName: \"kubernetes.io/projected/2deaa695-e9d6-4f83-b733-4125debcd73e-kube-api-access-mh46v\") pod \"2deaa695-e9d6-4f83-b733-4125debcd73e\" (UID: \"2deaa695-e9d6-4f83-b733-4125debcd73e\") "
Jul 29 22:59:17.493214 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:17.493186    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2deaa695-e9d6-4f83-b733-4125debcd73e-kube-api-access-mh46v" (OuterVolumeSpecName: "kube-api-access-mh46v") pod "2deaa695-e9d6-4f83-b733-4125debcd73e" (UID: "2deaa695-e9d6-4f83-b733-4125debcd73e"). InnerVolumeSpecName "kube-api-access-mh46v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:59:17.591833 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:17.591794    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh46v\" (UniqueName: \"kubernetes.io/projected/2deaa695-e9d6-4f83-b733-4125debcd73e-kube-api-access-mh46v\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 22:59:18.162452 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:18.162409    2569 generic.go:356] "Generic (PLEG): container finished" podID="2deaa695-e9d6-4f83-b733-4125debcd73e" containerID="23598ad4a3736c276a6930fc331556adb2e8e9076b27871ca5110ee737030e2b" exitCode=0
Jul 29 22:59:18.162452 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:18.162457    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-c4njl"
Jul 29 22:59:18.163009 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:18.162506    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-c4njl" event={"ID":"2deaa695-e9d6-4f83-b733-4125debcd73e","Type":"ContainerDied","Data":"23598ad4a3736c276a6930fc331556adb2e8e9076b27871ca5110ee737030e2b"}
Jul 29 22:59:18.163009 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:18.162551    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-c4njl" event={"ID":"2deaa695-e9d6-4f83-b733-4125debcd73e","Type":"ContainerDied","Data":"843f33a122c149464c35a170189442db76bf9f62b910b81c70ee147479e6ba69"}
Jul 29 22:59:18.163009 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:18.162569    2569 scope.go:117] "RemoveContainer" containerID="23598ad4a3736c276a6930fc331556adb2e8e9076b27871ca5110ee737030e2b"
Jul 29 22:59:18.171098 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:18.171076    2569 scope.go:117] "RemoveContainer" containerID="23598ad4a3736c276a6930fc331556adb2e8e9076b27871ca5110ee737030e2b"
Jul 29 22:59:18.171416 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:59:18.171393    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23598ad4a3736c276a6930fc331556adb2e8e9076b27871ca5110ee737030e2b\": container with ID starting with 23598ad4a3736c276a6930fc331556adb2e8e9076b27871ca5110ee737030e2b not found: ID does not exist" containerID="23598ad4a3736c276a6930fc331556adb2e8e9076b27871ca5110ee737030e2b"
Jul 29 22:59:18.171462 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:18.171425    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23598ad4a3736c276a6930fc331556adb2e8e9076b27871ca5110ee737030e2b"} err="failed to get container status \"23598ad4a3736c276a6930fc331556adb2e8e9076b27871ca5110ee737030e2b\": rpc error: code = NotFound desc = could not find container \"23598ad4a3736c276a6930fc331556adb2e8e9076b27871ca5110ee737030e2b\": container with ID starting with 23598ad4a3736c276a6930fc331556adb2e8e9076b27871ca5110ee737030e2b not found: ID does not exist"
Jul 29 22:59:18.190865 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:18.190831    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-c4njl"]
Jul 29 22:59:18.194341 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:18.194316    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-c4njl"]
Jul 29 22:59:19.473544 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:19.473511    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2deaa695-e9d6-4f83-b733-4125debcd73e" path="/var/lib/kubelet/pods/2deaa695-e9d6-4f83-b733-4125debcd73e/volumes"
Jul 29 22:59:35.958124 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:35.958089    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-xvmlj"]
Jul 29 22:59:35.958532 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:35.958269    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-xvmlj" podUID="c601d8ad-d8f6-4a5c-8508-daea32ddd5b8" containerName="authorino" containerID="cri-o://9747ae3a1c931f3dfc9912b4db1896392bc39417429baeabf3f849b9a966ab98" gracePeriod=30
Jul 29 22:59:36.207240 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:36.207211    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-xvmlj"
Jul 29 22:59:36.232854 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:36.232822    2569 generic.go:356] "Generic (PLEG): container finished" podID="c601d8ad-d8f6-4a5c-8508-daea32ddd5b8" containerID="9747ae3a1c931f3dfc9912b4db1896392bc39417429baeabf3f849b9a966ab98" exitCode=0
Jul 29 22:59:36.233044 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:36.232865    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-xvmlj"
Jul 29 22:59:36.233044 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:36.232899    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-xvmlj" event={"ID":"c601d8ad-d8f6-4a5c-8508-daea32ddd5b8","Type":"ContainerDied","Data":"9747ae3a1c931f3dfc9912b4db1896392bc39417429baeabf3f849b9a966ab98"}
Jul 29 22:59:36.233044 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:36.232948    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-xvmlj" event={"ID":"c601d8ad-d8f6-4a5c-8508-daea32ddd5b8","Type":"ContainerDied","Data":"628c64ac2e43273d04aa2128b0e06213ff856d3094d000241f3adc0de2255fb7"}
Jul 29 22:59:36.233044 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:36.232971    2569 scope.go:117] "RemoveContainer" containerID="9747ae3a1c931f3dfc9912b4db1896392bc39417429baeabf3f849b9a966ab98"
Jul 29 22:59:36.240876 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:36.240857    2569 scope.go:117] "RemoveContainer" containerID="9747ae3a1c931f3dfc9912b4db1896392bc39417429baeabf3f849b9a966ab98"
Jul 29 22:59:36.241175 ip-10-0-135-98 kubenswrapper[2569]: E0729 22:59:36.241153    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9747ae3a1c931f3dfc9912b4db1896392bc39417429baeabf3f849b9a966ab98\": container with ID starting with 9747ae3a1c931f3dfc9912b4db1896392bc39417429baeabf3f849b9a966ab98 not found: ID does not exist" containerID="9747ae3a1c931f3dfc9912b4db1896392bc39417429baeabf3f849b9a966ab98"
Jul 29 22:59:36.241243 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:36.241183    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9747ae3a1c931f3dfc9912b4db1896392bc39417429baeabf3f849b9a966ab98"} err="failed to get container status \"9747ae3a1c931f3dfc9912b4db1896392bc39417429baeabf3f849b9a966ab98\": rpc error: code = NotFound desc = could not find container \"9747ae3a1c931f3dfc9912b4db1896392bc39417429baeabf3f849b9a966ab98\": container with ID starting with 9747ae3a1c931f3dfc9912b4db1896392bc39417429baeabf3f849b9a966ab98 not found: ID does not exist"
Jul 29 22:59:36.359761 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:36.359720    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khnhg\" (UniqueName: \"kubernetes.io/projected/c601d8ad-d8f6-4a5c-8508-daea32ddd5b8-kube-api-access-khnhg\") pod \"c601d8ad-d8f6-4a5c-8508-daea32ddd5b8\" (UID: \"c601d8ad-d8f6-4a5c-8508-daea32ddd5b8\") "
Jul 29 22:59:36.361769 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:36.361736    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c601d8ad-d8f6-4a5c-8508-daea32ddd5b8-kube-api-access-khnhg" (OuterVolumeSpecName: "kube-api-access-khnhg") pod "c601d8ad-d8f6-4a5c-8508-daea32ddd5b8" (UID: "c601d8ad-d8f6-4a5c-8508-daea32ddd5b8"). InnerVolumeSpecName "kube-api-access-khnhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:59:36.460445 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:36.460403    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khnhg\" (UniqueName: \"kubernetes.io/projected/c601d8ad-d8f6-4a5c-8508-daea32ddd5b8-kube-api-access-khnhg\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 22:59:36.553285 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:36.553245    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-xvmlj"]
Jul 29 22:59:36.556584 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:36.556556    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-xvmlj"]
Jul 29 22:59:37.473283 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:37.473247    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c601d8ad-d8f6-4a5c-8508-daea32ddd5b8" path="/var/lib/kubelet/pods/c601d8ad-d8f6-4a5c-8508-daea32ddd5b8/volumes"
Jul 29 22:59:40.424143 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.424092    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["observability/jaeger-6bbd7d4c9d-lgvjv"]
Jul 29 22:59:40.424559 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.424394    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2deaa695-e9d6-4f83-b733-4125debcd73e" containerName="authorino"
Jul 29 22:59:40.424559 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.424408    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2deaa695-e9d6-4f83-b733-4125debcd73e" containerName="authorino"
Jul 29 22:59:40.424559 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.424420    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c601d8ad-d8f6-4a5c-8508-daea32ddd5b8" containerName="authorino"
Jul 29 22:59:40.424559 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.424426    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c601d8ad-d8f6-4a5c-8508-daea32ddd5b8" containerName="authorino"
Jul 29 22:59:40.424559 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.424497    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2deaa695-e9d6-4f83-b733-4125debcd73e" containerName="authorino"
Jul 29 22:59:40.424559 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.424505    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c601d8ad-d8f6-4a5c-8508-daea32ddd5b8" containerName="authorino"
Jul 29 22:59:40.428745 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.428726    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-lgvjv"
Jul 29 22:59:40.431391 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.431374    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"openshift-service-ca.crt\""
Jul 29 22:59:40.431978 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.431964    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"observability\"/\"jaeger-dockercfg-w4r5d\""
Jul 29 22:59:40.432034 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.431976    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"kube-root-ca.crt\""
Jul 29 22:59:40.438131 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.438105    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-lgvjv"]
Jul 29 22:59:40.486417 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.486379    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f8v7\" (UniqueName: \"kubernetes.io/projected/8ddca28b-f4c5-4d4f-a432-224bb57fdbce-kube-api-access-7f8v7\") pod \"jaeger-6bbd7d4c9d-lgvjv\" (UID: \"8ddca28b-f4c5-4d4f-a432-224bb57fdbce\") " pod="observability/jaeger-6bbd7d4c9d-lgvjv"
Jul 29 22:59:40.586934 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.586895    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f8v7\" (UniqueName: \"kubernetes.io/projected/8ddca28b-f4c5-4d4f-a432-224bb57fdbce-kube-api-access-7f8v7\") pod \"jaeger-6bbd7d4c9d-lgvjv\" (UID: \"8ddca28b-f4c5-4d4f-a432-224bb57fdbce\") " pod="observability/jaeger-6bbd7d4c9d-lgvjv"
Jul 29 22:59:40.594895 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.594864    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f8v7\" (UniqueName: \"kubernetes.io/projected/8ddca28b-f4c5-4d4f-a432-224bb57fdbce-kube-api-access-7f8v7\") pod \"jaeger-6bbd7d4c9d-lgvjv\" (UID: \"8ddca28b-f4c5-4d4f-a432-224bb57fdbce\") " pod="observability/jaeger-6bbd7d4c9d-lgvjv"
Jul 29 22:59:40.738143 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.738111    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-lgvjv"
Jul 29 22:59:40.867726 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:40.867697    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-lgvjv"]
Jul 29 22:59:40.870506 ip-10-0-135-98 kubenswrapper[2569]: W0729 22:59:40.870447    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ddca28b_f4c5_4d4f_a432_224bb57fdbce.slice/crio-a1faf7e5aa1639bc35426d82478f6ff7a48ca9ff70243bd2f297aae9b568a7fe WatchSource:0}: Error finding container a1faf7e5aa1639bc35426d82478f6ff7a48ca9ff70243bd2f297aae9b568a7fe: Status 404 returned error can't find the container with id a1faf7e5aa1639bc35426d82478f6ff7a48ca9ff70243bd2f297aae9b568a7fe
Jul 29 22:59:41.254011 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:41.253974    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-lgvjv" event={"ID":"8ddca28b-f4c5-4d4f-a432-224bb57fdbce","Type":"ContainerStarted","Data":"a1faf7e5aa1639bc35426d82478f6ff7a48ca9ff70243bd2f297aae9b568a7fe"}
Jul 29 22:59:43.263151 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:43.263113    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-lgvjv" event={"ID":"8ddca28b-f4c5-4d4f-a432-224bb57fdbce","Type":"ContainerStarted","Data":"2594ef4251278129f6004d2d16c4e986dce999c96ee225d8fa32566d2de5b667"}
Jul 29 22:59:43.263603 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:43.263315    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="observability/jaeger-6bbd7d4c9d-lgvjv"
Jul 29 22:59:43.265671 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:43.265650    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="observability/jaeger-6bbd7d4c9d-lgvjv"
Jul 29 22:59:43.278497 ip-10-0-135-98 kubenswrapper[2569]: I0729 22:59:43.278425    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="observability/jaeger-6bbd7d4c9d-lgvjv" podStartSLOduration=1.533852454 podStartE2EDuration="3.27840851s" totalImagesPullingTime="1.744556056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:59:43.27733436 +0000 UTC m=+790.282531785" watchObservedRunningTime="2026-07-29 22:59:43.27840851 +0000 UTC m=+790.283605943"
Jul 29 23:00:36.041364 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:36.041330    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-mhzx7"]
Jul 29 23:00:36.045062 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:36.045038    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mhzx7"
Jul 29 23:00:36.047522 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:36.047500    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-5sh48\""
Jul 29 23:00:36.047653 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:36.047502    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 29 23:00:36.052919 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:36.052897    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-mhzx7"]
Jul 29 23:00:36.054501 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:36.054454    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p75kx\" (UniqueName: \"kubernetes.io/projected/1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1-kube-api-access-p75kx\") pod \"s3-init-mhzx7\" (UID: \"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1\") " pod="kserve/s3-init-mhzx7"
Jul 29 23:00:36.054601 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:36.054500    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1-model-cache\") pod \"s3-init-mhzx7\" (UID: \"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1\") " pod="kserve/s3-init-mhzx7"
Jul 29 23:00:36.155346 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:36.155304    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p75kx\" (UniqueName: \"kubernetes.io/projected/1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1-kube-api-access-p75kx\") pod \"s3-init-mhzx7\" (UID: \"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1\") " pod="kserve/s3-init-mhzx7"
Jul 29 23:00:36.155346 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:36.155346    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1-model-cache\") pod \"s3-init-mhzx7\" (UID: \"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1\") " pod="kserve/s3-init-mhzx7"
Jul 29 23:00:36.155738 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:36.155718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1-model-cache\") pod \"s3-init-mhzx7\" (UID: \"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1\") " pod="kserve/s3-init-mhzx7"
Jul 29 23:00:36.163714 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:36.163688    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p75kx\" (UniqueName: \"kubernetes.io/projected/1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1-kube-api-access-p75kx\") pod \"s3-init-mhzx7\" (UID: \"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1\") " pod="kserve/s3-init-mhzx7"
Jul 29 23:00:36.354598 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:36.354513    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mhzx7"
Jul 29 23:00:36.486343 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:36.486309    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-mhzx7"]
Jul 29 23:00:36.489979 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:00:36.489948    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1145b1ed_c80c_4b4a_b3a8_bf3f98ff8ba1.slice/crio-661ea4f48f238e3102ef0a6b543919f227ab11031fbec8f22ee1668202674e92 WatchSource:0}: Error finding container 661ea4f48f238e3102ef0a6b543919f227ab11031fbec8f22ee1668202674e92: Status 404 returned error can't find the container with id 661ea4f48f238e3102ef0a6b543919f227ab11031fbec8f22ee1668202674e92
Jul 29 23:00:37.474362 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:37.474325    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mhzx7" event={"ID":"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1","Type":"ContainerStarted","Data":"661ea4f48f238e3102ef0a6b543919f227ab11031fbec8f22ee1668202674e92"}
Jul 29 23:00:41.491247 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:41.491209    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mhzx7" event={"ID":"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1","Type":"ContainerStarted","Data":"653bffbe85abed3ec56842932336166cfa8649cd571297dc3b3c8524c1ea6c53"}
Jul 29 23:00:43.503226 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:43.503189    2569 generic.go:356] "Generic (PLEG): container finished" podID="1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1" containerID="653bffbe85abed3ec56842932336166cfa8649cd571297dc3b3c8524c1ea6c53" exitCode=0
Jul 29 23:00:43.503636 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:43.503253    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mhzx7" event={"ID":"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1","Type":"ContainerDied","Data":"653bffbe85abed3ec56842932336166cfa8649cd571297dc3b3c8524c1ea6c53"}
Jul 29 23:00:48.529523 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:48.529467    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mhzx7" event={"ID":"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1","Type":"ContainerStarted","Data":"94cf4a83eabede7edfcc1024f20c59fd46fb02389efce425741bf945cf88ba90"}
Jul 29 23:00:48.546585 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:00:48.546536    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-mhzx7" podStartSLOduration=1.5811327880000001 podStartE2EDuration="12.546521157s" totalImagesPullingTime="8.72278788s" totalInitContainerRuntime="2.242600489s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:00:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:00:48.545313299 +0000 UTC m=+855.550510726" watchObservedRunningTime="2026-07-29 23:00:48.546521157 +0000 UTC m=+855.551718579"
Jul 29 23:01:33.463429 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:01:33.463395    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:01:33.464297 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:01:33.464274    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:01:53.775808 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:01:53.775771    2569 generic.go:356] "Generic (PLEG): container finished" podID="1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1" containerID="94cf4a83eabede7edfcc1024f20c59fd46fb02389efce425741bf945cf88ba90" exitCode=0
Jul 29 23:01:53.776254 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:01:53.775855    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mhzx7" event={"ID":"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1","Type":"ContainerDied","Data":"94cf4a83eabede7edfcc1024f20c59fd46fb02389efce425741bf945cf88ba90"}
Jul 29 23:01:54.914291 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:01:54.914259    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mhzx7"
Jul 29 23:01:55.019411 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:01:55.019369    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1-model-cache\") pod \"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1\" (UID: \"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1\") "
Jul 29 23:01:55.019632 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:01:55.019477    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p75kx\" (UniqueName: \"kubernetes.io/projected/1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1-kube-api-access-p75kx\") pod \"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1\" (UID: \"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1\") "
Jul 29 23:01:55.021763 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:01:55.021726    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1-kube-api-access-p75kx" (OuterVolumeSpecName: "kube-api-access-p75kx") pod "1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1" (UID: "1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1"). InnerVolumeSpecName "kube-api-access-p75kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:01:55.038420 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:01:55.038337    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1-model-cache" (OuterVolumeSpecName: "model-cache") pod "1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1" (UID: "1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:01:55.120188 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:01:55.120155    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p75kx\" (UniqueName: \"kubernetes.io/projected/1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1-kube-api-access-p75kx\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:01:55.120188 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:01:55.120184    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:01:55.785329 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:01:55.785251    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mhzx7"
Jul 29 23:01:55.785512 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:01:55.785258    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mhzx7" event={"ID":"1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1","Type":"ContainerDied","Data":"661ea4f48f238e3102ef0a6b543919f227ab11031fbec8f22ee1668202674e92"}
Jul 29 23:01:55.785633 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:01:55.785531    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="661ea4f48f238e3102ef0a6b543919f227ab11031fbec8f22ee1668202674e92"
Jul 29 23:02:22.181036 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.180988    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"]
Jul 29 23:02:22.181726 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.181422    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1" containerName="s3-init"
Jul 29 23:02:22.181726 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.181438    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1" containerName="s3-init"
Jul 29 23:02:22.181726 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.181451    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1" containerName="download-hf-model"
Jul 29 23:02:22.181726 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.181460    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1" containerName="download-hf-model"
Jul 29 23:02:22.181726 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.181573    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1" containerName="s3-init"
Jul 29 23:02:22.184801 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.184773    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.187425 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.187390    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-fc8tf\""
Jul 29 23:02:22.187425 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.187415    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 29 23:02:22.187654 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.187444    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 29 23:02:22.187654 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.187390    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 29 23:02:22.194547 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.194520    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"]
Jul 29 23:02:22.254421 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.254382    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.254421 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.254426    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-dshm\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.254654 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.254530    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx5gx\" (UniqueName: \"kubernetes.io/projected/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-kube-api-access-fx5gx\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.254654 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.254564    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.254654 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.254584    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-home\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.254750 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.254669    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.355601 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.355571    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.355776 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.355610    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-dshm\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.355776 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.355741    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx5gx\" (UniqueName: \"kubernetes.io/projected/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-kube-api-access-fx5gx\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.355879 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.355777    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.355879 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.355799    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-home\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.355989 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.355931    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.356040 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.356023    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.356094 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.356076    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-home\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.356223 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.356202    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.357775 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.357747    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-dshm\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.358126 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.358103    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.363304 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.363285    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx5gx\" (UniqueName: \"kubernetes.io/projected/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-kube-api-access-fx5gx\") pod \"fc-smoke-test-kserve-7f4687897-tmtnm\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.497401 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.497362    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:22.632959 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.632914    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"]
Jul 29 23:02:22.636130 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:02:22.636092    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e9a016_ed91_44e7_85e2_50df14ab9ea9.slice/crio-111a06a230024b08e08b7bf68ebe52ddad32b9e30ac32dd6c7e5cf5d05b4a123 WatchSource:0}: Error finding container 111a06a230024b08e08b7bf68ebe52ddad32b9e30ac32dd6c7e5cf5d05b4a123: Status 404 returned error can't find the container with id 111a06a230024b08e08b7bf68ebe52ddad32b9e30ac32dd6c7e5cf5d05b4a123
Jul 29 23:02:22.638434 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.638417    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 23:02:22.893018 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:22.892932    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm" event={"ID":"b1e9a016-ed91-44e7-85e2-50df14ab9ea9","Type":"ContainerStarted","Data":"111a06a230024b08e08b7bf68ebe52ddad32b9e30ac32dd6c7e5cf5d05b4a123"}
Jul 29 23:02:24.902867 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:24.902830    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm" event={"ID":"b1e9a016-ed91-44e7-85e2-50df14ab9ea9","Type":"ContainerStarted","Data":"522ac7706bec9906e6712048d1f146a9053689f04f1961c9061e35e92ee6a226"}
Jul 29 23:02:24.919377 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:24.919326    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm" podStartSLOduration=1.419720814 podStartE2EDuration="2.919310295s" totalImagesPullingTime="1.499589481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:02:24.918337286 +0000 UTC m=+951.923534719" watchObservedRunningTime="2026-07-29 23:02:24.919310295 +0000 UTC m=+951.924507718"
Jul 29 23:02:25.498187 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:25.498154    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:32.498558 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:32.498520    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:32.510901 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:32.510870    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:02:32.950649 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:02:32.950566    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:03:14.319842 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.319762    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"]
Jul 29 23:03:14.328751 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:03:14.328697    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:03:14.328751 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:03:14.328751 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:03:14.328751 ip-10-0-135-98 kubenswrapper[2569]:  > containerID="522ac7706bec9906e6712048d1f146a9053689f04f1961c9061e35e92ee6a226" cmd=["/bin/sleep","15"]
Jul 29 23:03:14.329046 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.328774    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm" message="" err=<
Jul 29 23:03:14.329046 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:03:14.329046 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:03:14.329046 ip-10-0-135-98 kubenswrapper[2569]:  >
Jul 29 23:03:14.329046 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:03:14.328791    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:03:14.329046 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:03:14.329046 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:03:14.329046 ip-10-0-135-98 kubenswrapper[2569]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm" podUID="b1e9a016-ed91-44e7-85e2-50df14ab9ea9" containerName="main" containerID="cri-o://522ac7706bec9906e6712048d1f146a9053689f04f1961c9061e35e92ee6a226"
Jul 29 23:03:14.329046 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.328851    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm" podUID="b1e9a016-ed91-44e7-85e2-50df14ab9ea9" containerName="main" containerID="cri-o://522ac7706bec9906e6712048d1f146a9053689f04f1961c9061e35e92ee6a226" gracePeriod=60
Jul 29 23:03:14.686655 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.686626    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:03:14.792121 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.792078    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-tmp-dir\") pod \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") "
Jul 29 23:03:14.792121 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.792128    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-model-cache\") pod \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") "
Jul 29 23:03:14.792344 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.792167    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx5gx\" (UniqueName: \"kubernetes.io/projected/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-kube-api-access-fx5gx\") pod \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") "
Jul 29 23:03:14.792344 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.792185    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-dshm\") pod \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") "
Jul 29 23:03:14.792344 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.792249    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-tls-certs\") pod \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") "
Jul 29 23:03:14.792344 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.792304    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-home\") pod \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\" (UID: \"b1e9a016-ed91-44e7-85e2-50df14ab9ea9\") "
Jul 29 23:03:14.792578 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.792404    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1e9a016-ed91-44e7-85e2-50df14ab9ea9" (UID: "b1e9a016-ed91-44e7-85e2-50df14ab9ea9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:03:14.792578 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.792467    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1e9a016-ed91-44e7-85e2-50df14ab9ea9" (UID: "b1e9a016-ed91-44e7-85e2-50df14ab9ea9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:03:14.792578 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.792561    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:03:14.792578 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.792574    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:03:14.792808 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.792670    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-home" (OuterVolumeSpecName: "home") pod "b1e9a016-ed91-44e7-85e2-50df14ab9ea9" (UID: "b1e9a016-ed91-44e7-85e2-50df14ab9ea9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:03:14.794513 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.794467    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-dshm" (OuterVolumeSpecName: "dshm") pod "b1e9a016-ed91-44e7-85e2-50df14ab9ea9" (UID: "b1e9a016-ed91-44e7-85e2-50df14ab9ea9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:03:14.794628 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.794462    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1e9a016-ed91-44e7-85e2-50df14ab9ea9" (UID: "b1e9a016-ed91-44e7-85e2-50df14ab9ea9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:03:14.794628 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.794472    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-kube-api-access-fx5gx" (OuterVolumeSpecName: "kube-api-access-fx5gx") pod "b1e9a016-ed91-44e7-85e2-50df14ab9ea9" (UID: "b1e9a016-ed91-44e7-85e2-50df14ab9ea9"). InnerVolumeSpecName "kube-api-access-fx5gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:03:14.893118 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.893027    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:03:14.893118 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.893064    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx5gx\" (UniqueName: \"kubernetes.io/projected/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-kube-api-access-fx5gx\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:03:14.893118 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.893074    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:03:14.893118 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:14.893082    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e9a016-ed91-44e7-85e2-50df14ab9ea9-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:03:15.107007 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:15.106970    2569 generic.go:356] "Generic (PLEG): container finished" podID="b1e9a016-ed91-44e7-85e2-50df14ab9ea9" containerID="522ac7706bec9906e6712048d1f146a9053689f04f1961c9061e35e92ee6a226" exitCode=0
Jul 29 23:03:15.107189 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:15.107018    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"
Jul 29 23:03:15.107189 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:15.107056    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm" event={"ID":"b1e9a016-ed91-44e7-85e2-50df14ab9ea9","Type":"ContainerDied","Data":"522ac7706bec9906e6712048d1f146a9053689f04f1961c9061e35e92ee6a226"}
Jul 29 23:03:15.107189 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:15.107120    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm" event={"ID":"b1e9a016-ed91-44e7-85e2-50df14ab9ea9","Type":"ContainerDied","Data":"111a06a230024b08e08b7bf68ebe52ddad32b9e30ac32dd6c7e5cf5d05b4a123"}
Jul 29 23:03:15.107189 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:15.107150    2569 scope.go:117] "RemoveContainer" containerID="522ac7706bec9906e6712048d1f146a9053689f04f1961c9061e35e92ee6a226"
Jul 29 23:03:15.116320 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:15.116303    2569 scope.go:117] "RemoveContainer" containerID="522ac7706bec9906e6712048d1f146a9053689f04f1961c9061e35e92ee6a226"
Jul 29 23:03:15.116615 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:03:15.116595    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522ac7706bec9906e6712048d1f146a9053689f04f1961c9061e35e92ee6a226\": container with ID starting with 522ac7706bec9906e6712048d1f146a9053689f04f1961c9061e35e92ee6a226 not found: ID does not exist" containerID="522ac7706bec9906e6712048d1f146a9053689f04f1961c9061e35e92ee6a226"
Jul 29 23:03:15.116712 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:15.116622    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522ac7706bec9906e6712048d1f146a9053689f04f1961c9061e35e92ee6a226"} err="failed to get container status \"522ac7706bec9906e6712048d1f146a9053689f04f1961c9061e35e92ee6a226\": rpc error: code = NotFound desc = could not find container \"522ac7706bec9906e6712048d1f146a9053689f04f1961c9061e35e92ee6a226\": container with ID starting with 522ac7706bec9906e6712048d1f146a9053689f04f1961c9061e35e92ee6a226 not found: ID does not exist"
Jul 29 23:03:15.127641 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:15.127615    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"]
Jul 29 23:03:15.134064 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:15.134043    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tmtnm"]
Jul 29 23:03:15.474226 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:15.474191    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1e9a016-ed91-44e7-85e2-50df14ab9ea9" path="/var/lib/kubelet/pods/b1e9a016-ed91-44e7-85e2-50df14ab9ea9/volumes"
Jul 29 23:03:40.389799 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.389765    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"]
Jul 29 23:03:40.390216 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.390048    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e9a016-ed91-44e7-85e2-50df14ab9ea9" containerName="main"
Jul 29 23:03:40.390216 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.390059    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e9a016-ed91-44e7-85e2-50df14ab9ea9" containerName="main"
Jul 29 23:03:40.390216 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.390113    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e9a016-ed91-44e7-85e2-50df14ab9ea9" containerName="main"
Jul 29 23:03:40.394159 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.394138    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.396638 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.396614    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 29 23:03:40.396760 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.396661    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 29 23:03:40.396760 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.396666    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-8pprj\""
Jul 29 23:03:40.396760 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.396670    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 29 23:03:40.403544 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.403523    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"]
Jul 29 23:03:40.513239 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.513206    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a635f0-4de3-4543-94c1-627259174833-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.513239 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.513242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.513448 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.513264    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-home\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.513448 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.513280    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.513448 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.513358    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.513448 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.513432    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvxst\" (UniqueName: \"kubernetes.io/projected/28a635f0-4de3-4543-94c1-627259174833-kube-api-access-mvxst\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.614061 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.614024    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-home\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.614061 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.614061    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.614289 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.614084    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.614289 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.614162    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvxst\" (UniqueName: \"kubernetes.io/projected/28a635f0-4de3-4543-94c1-627259174833-kube-api-access-mvxst\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.614532 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.614507    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-home\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.614631 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.614538    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.614631 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.614538    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a635f0-4de3-4543-94c1-627259174833-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.614631 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.614579    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.614631 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.614621    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.616803 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.616777    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.616914 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.616898    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a635f0-4de3-4543-94c1-627259174833-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.623467 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.623413    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvxst\" (UniqueName: \"kubernetes.io/projected/28a635f0-4de3-4543-94c1-627259174833-kube-api-access-mvxst\") pod \"fc-concurrency-test-kserve-b5d44467c-tdvzz\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.704927 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.704836    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:40.838096 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:40.837991    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"]
Jul 29 23:03:40.840958 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:03:40.840928    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a635f0_4de3_4543_94c1_627259174833.slice/crio-7e079e1b9a96e6d14feea94d49c7073cbb945fa5beadb1d2130ca9c95a5e7fc5 WatchSource:0}: Error finding container 7e079e1b9a96e6d14feea94d49c7073cbb945fa5beadb1d2130ca9c95a5e7fc5: Status 404 returned error can't find the container with id 7e079e1b9a96e6d14feea94d49c7073cbb945fa5beadb1d2130ca9c95a5e7fc5
Jul 29 23:03:41.206816 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:41.206780    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz" event={"ID":"28a635f0-4de3-4543-94c1-627259174833","Type":"ContainerStarted","Data":"d4137f272fbef73c357d288c5bb1617dda01f8bd855dcc26350602dc80d0ca65"}
Jul 29 23:03:41.207015 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:41.206835    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz" event={"ID":"28a635f0-4de3-4543-94c1-627259174833","Type":"ContainerStarted","Data":"7e079e1b9a96e6d14feea94d49c7073cbb945fa5beadb1d2130ca9c95a5e7fc5"}
Jul 29 23:03:41.223533 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:41.223386    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz" podStartSLOduration=1.22336799 podStartE2EDuration="1.22336799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:03:41.222828457 +0000 UTC m=+1028.228025880" watchObservedRunningTime="2026-07-29 23:03:41.22336799 +0000 UTC m=+1028.228565413"
Jul 29 23:03:41.705266 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:41.705213    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:50.705179 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:50.705140    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:50.718062 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:50.718035    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:03:51.256112 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:03:51.256081    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:04:19.323738 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.323700    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"]
Jul 29 23:04:19.336336 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:04:19.336295    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:04:19.336336 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:04:19.336336 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:04:19.336336 ip-10-0-135-98 kubenswrapper[2569]:  > containerID="d4137f272fbef73c357d288c5bb1617dda01f8bd855dcc26350602dc80d0ca65" cmd=["/bin/sleep","15"]
Jul 29 23:04:19.336658 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.336355    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz" message="" err=<
Jul 29 23:04:19.336658 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:04:19.336658 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:04:19.336658 ip-10-0-135-98 kubenswrapper[2569]:  >
Jul 29 23:04:19.336658 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:04:19.336379    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:04:19.336658 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:04:19.336658 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:04:19.336658 ip-10-0-135-98 kubenswrapper[2569]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz" podUID="28a635f0-4de3-4543-94c1-627259174833" containerName="main" containerID="cri-o://d4137f272fbef73c357d288c5bb1617dda01f8bd855dcc26350602dc80d0ca65"
Jul 29 23:04:19.336658 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.336421    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz" podUID="28a635f0-4de3-4543-94c1-627259174833" containerName="main" containerID="cri-o://d4137f272fbef73c357d288c5bb1617dda01f8bd855dcc26350602dc80d0ca65" gracePeriod=60
Jul 29 23:04:19.685680 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.685654    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:04:19.740600 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.740572    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a635f0-4de3-4543-94c1-627259174833-tls-certs\") pod \"28a635f0-4de3-4543-94c1-627259174833\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") "
Jul 29 23:04:19.740771 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.740622    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-model-cache\") pod \"28a635f0-4de3-4543-94c1-627259174833\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") "
Jul 29 23:04:19.740771 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.740656    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-tmp-dir\") pod \"28a635f0-4de3-4543-94c1-627259174833\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") "
Jul 29 23:04:19.740771 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.740700    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvxst\" (UniqueName: \"kubernetes.io/projected/28a635f0-4de3-4543-94c1-627259174833-kube-api-access-mvxst\") pod \"28a635f0-4de3-4543-94c1-627259174833\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") "
Jul 29 23:04:19.740771 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.740755    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-home\") pod \"28a635f0-4de3-4543-94c1-627259174833\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") "
Jul 29 23:04:19.740982 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.740832    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-dshm\") pod \"28a635f0-4de3-4543-94c1-627259174833\" (UID: \"28a635f0-4de3-4543-94c1-627259174833\") "
Jul 29 23:04:19.741035 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.740978    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-model-cache" (OuterVolumeSpecName: "model-cache") pod "28a635f0-4de3-4543-94c1-627259174833" (UID: "28a635f0-4de3-4543-94c1-627259174833"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:04:19.741103 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.741030    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28a635f0-4de3-4543-94c1-627259174833" (UID: "28a635f0-4de3-4543-94c1-627259174833"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:04:19.741103 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.741087    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-home" (OuterVolumeSpecName: "home") pod "28a635f0-4de3-4543-94c1-627259174833" (UID: "28a635f0-4de3-4543-94c1-627259174833"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:04:19.742867 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.742836    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a635f0-4de3-4543-94c1-627259174833-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28a635f0-4de3-4543-94c1-627259174833" (UID: "28a635f0-4de3-4543-94c1-627259174833"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:04:19.742971 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.742875    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-dshm" (OuterVolumeSpecName: "dshm") pod "28a635f0-4de3-4543-94c1-627259174833" (UID: "28a635f0-4de3-4543-94c1-627259174833"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:04:19.742971 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.742936    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a635f0-4de3-4543-94c1-627259174833-kube-api-access-mvxst" (OuterVolumeSpecName: "kube-api-access-mvxst") pod "28a635f0-4de3-4543-94c1-627259174833" (UID: "28a635f0-4de3-4543-94c1-627259174833"). InnerVolumeSpecName "kube-api-access-mvxst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:04:19.841652 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.841617    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:04:19.841652 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.841646    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:04:19.841652 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.841655    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a635f0-4de3-4543-94c1-627259174833-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:04:19.841873 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.841668    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:04:19.841873 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.841697    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a635f0-4de3-4543-94c1-627259174833-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:04:19.841873 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:19.841706    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvxst\" (UniqueName: \"kubernetes.io/projected/28a635f0-4de3-4543-94c1-627259174833-kube-api-access-mvxst\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:04:20.362696 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:20.362662    2569 generic.go:356] "Generic (PLEG): container finished" podID="28a635f0-4de3-4543-94c1-627259174833" containerID="d4137f272fbef73c357d288c5bb1617dda01f8bd855dcc26350602dc80d0ca65" exitCode=0
Jul 29 23:04:20.363129 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:20.362713    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"
Jul 29 23:04:20.363129 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:20.362750    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz" event={"ID":"28a635f0-4de3-4543-94c1-627259174833","Type":"ContainerDied","Data":"d4137f272fbef73c357d288c5bb1617dda01f8bd855dcc26350602dc80d0ca65"}
Jul 29 23:04:20.363129 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:20.362807    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz" event={"ID":"28a635f0-4de3-4543-94c1-627259174833","Type":"ContainerDied","Data":"7e079e1b9a96e6d14feea94d49c7073cbb945fa5beadb1d2130ca9c95a5e7fc5"}
Jul 29 23:04:20.363129 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:20.362829    2569 scope.go:117] "RemoveContainer" containerID="d4137f272fbef73c357d288c5bb1617dda01f8bd855dcc26350602dc80d0ca65"
Jul 29 23:04:20.372395 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:20.372380    2569 scope.go:117] "RemoveContainer" containerID="d4137f272fbef73c357d288c5bb1617dda01f8bd855dcc26350602dc80d0ca65"
Jul 29 23:04:20.372696 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:04:20.372678    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4137f272fbef73c357d288c5bb1617dda01f8bd855dcc26350602dc80d0ca65\": container with ID starting with d4137f272fbef73c357d288c5bb1617dda01f8bd855dcc26350602dc80d0ca65 not found: ID does not exist" containerID="d4137f272fbef73c357d288c5bb1617dda01f8bd855dcc26350602dc80d0ca65"
Jul 29 23:04:20.372780 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:20.372704    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4137f272fbef73c357d288c5bb1617dda01f8bd855dcc26350602dc80d0ca65"} err="failed to get container status \"d4137f272fbef73c357d288c5bb1617dda01f8bd855dcc26350602dc80d0ca65\": rpc error: code = NotFound desc = could not find container \"d4137f272fbef73c357d288c5bb1617dda01f8bd855dcc26350602dc80d0ca65\": container with ID starting with d4137f272fbef73c357d288c5bb1617dda01f8bd855dcc26350602dc80d0ca65 not found: ID does not exist"
Jul 29 23:04:20.383329 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:20.383305    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"]
Jul 29 23:04:20.386794 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:20.386776    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-tdvzz"]
Jul 29 23:04:21.474282 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:04:21.474233    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28a635f0-4de3-4543-94c1-627259174833" path="/var/lib/kubelet/pods/28a635f0-4de3-4543-94c1-627259174833/volumes"
Jul 29 23:05:00.290683 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.290648    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"]
Jul 29 23:05:00.291103 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.290959    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28a635f0-4de3-4543-94c1-627259174833" containerName="main"
Jul 29 23:05:00.291103 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.290971    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="28a635f0-4de3-4543-94c1-627259174833" containerName="main"
Jul 29 23:05:00.291103 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.291041    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="28a635f0-4de3-4543-94c1-627259174833" containerName="main"
Jul 29 23:05:00.295005 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.294983    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.297784 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.297757    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-j9zjb\""
Jul 29 23:05:00.297784 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.297778    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 29 23:05:00.297953 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.297758    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 29 23:05:00.297953 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.297762    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 29 23:05:00.303106 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.303085    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"]
Jul 29 23:05:00.486983 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.486947    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl52b\" (UniqueName: \"kubernetes.io/projected/7246b6cb-1311-42bc-9eff-d42cdb01ad38-kube-api-access-jl52b\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.487163 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.486994    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7246b6cb-1311-42bc-9eff-d42cdb01ad38-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.487163 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.487028    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.487163 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.487074    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.487163 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.487110    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.487163 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.487132    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.587936 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.587841    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl52b\" (UniqueName: \"kubernetes.io/projected/7246b6cb-1311-42bc-9eff-d42cdb01ad38-kube-api-access-jl52b\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.587936 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.587899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7246b6cb-1311-42bc-9eff-d42cdb01ad38-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.587936 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.587923    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.588219 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.587948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.588219 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.587977    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.588219 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.587997    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.588391 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.588366    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.588391 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.588381    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.588516 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.588434    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.590337 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.590316    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.590553 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.590536    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7246b6cb-1311-42bc-9eff-d42cdb01ad38-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.594834 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.594812    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl52b\" (UniqueName: \"kubernetes.io/projected/7246b6cb-1311-42bc-9eff-d42cdb01ad38-kube-api-access-jl52b\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.605638 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.605620    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:00.741191 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:00.741151    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"]
Jul 29 23:05:00.743732 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:05:00.743699    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7246b6cb_1311_42bc_9eff_d42cdb01ad38.slice/crio-85a0d8a2acac385a812724a47efd861e7c5145ad0d93a8dfa8f6244aa03f5562 WatchSource:0}: Error finding container 85a0d8a2acac385a812724a47efd861e7c5145ad0d93a8dfa8f6244aa03f5562: Status 404 returned error can't find the container with id 85a0d8a2acac385a812724a47efd861e7c5145ad0d93a8dfa8f6244aa03f5562
Jul 29 23:05:01.522950 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:01.522917    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl" event={"ID":"7246b6cb-1311-42bc-9eff-d42cdb01ad38","Type":"ContainerStarted","Data":"64632728be939bd3a9a4c07624cd96b1522d032f70c99ee8f69e7d709fd1f616"}
Jul 29 23:05:01.523332 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:01.522967    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl" event={"ID":"7246b6cb-1311-42bc-9eff-d42cdb01ad38","Type":"ContainerStarted","Data":"85a0d8a2acac385a812724a47efd861e7c5145ad0d93a8dfa8f6244aa03f5562"}
Jul 29 23:05:01.538934 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:01.538885    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl" podStartSLOduration=1.5388709010000001 podStartE2EDuration="1.538870901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:05:01.537697557 +0000 UTC m=+1108.542894981" watchObservedRunningTime="2026-07-29 23:05:01.538870901 +0000 UTC m=+1108.544068323"
Jul 29 23:05:01.606317 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:01.606287    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:03.668710 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:03.668677    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"]
Jul 29 23:05:04.548837 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:05:04.548798    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:05:04.548837 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:05:04.548837 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:05:04.548837 ip-10-0-135-98 kubenswrapper[2569]:  > containerID="64632728be939bd3a9a4c07624cd96b1522d032f70c99ee8f69e7d709fd1f616" cmd=["/bin/sleep","15"]
Jul 29 23:05:04.549138 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.548858    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl" message="" err=<
Jul 29 23:05:04.549138 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:05:04.549138 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:05:04.549138 ip-10-0-135-98 kubenswrapper[2569]:  >
Jul 29 23:05:04.549138 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:05:04.548874    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:05:04.549138 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:05:04.549138 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:05:04.549138 ip-10-0-135-98 kubenswrapper[2569]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl" podUID="7246b6cb-1311-42bc-9eff-d42cdb01ad38" containerName="main" containerID="cri-o://64632728be939bd3a9a4c07624cd96b1522d032f70c99ee8f69e7d709fd1f616"
Jul 29 23:05:04.549138 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.548920    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl" podUID="7246b6cb-1311-42bc-9eff-d42cdb01ad38" containerName="main" containerID="cri-o://64632728be939bd3a9a4c07624cd96b1522d032f70c99ee8f69e7d709fd1f616" gracePeriod=60
Jul 29 23:05:04.795520 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.795471    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:04.823953 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.823854    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-model-cache\") pod \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") "
Jul 29 23:05:04.823953 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.823896    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7246b6cb-1311-42bc-9eff-d42cdb01ad38-tls-certs\") pod \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") "
Jul 29 23:05:04.823953 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.823917    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-dshm\") pod \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") "
Jul 29 23:05:04.823953 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.823942    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-tmp-dir\") pod \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") "
Jul 29 23:05:04.824362 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.824001    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-home\") pod \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") "
Jul 29 23:05:04.824362 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.824031    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl52b\" (UniqueName: \"kubernetes.io/projected/7246b6cb-1311-42bc-9eff-d42cdb01ad38-kube-api-access-jl52b\") pod \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\" (UID: \"7246b6cb-1311-42bc-9eff-d42cdb01ad38\") "
Jul 29 23:05:04.824362 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.824207    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-model-cache" (OuterVolumeSpecName: "model-cache") pod "7246b6cb-1311-42bc-9eff-d42cdb01ad38" (UID: "7246b6cb-1311-42bc-9eff-d42cdb01ad38"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:04.824362 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.824288    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-home" (OuterVolumeSpecName: "home") pod "7246b6cb-1311-42bc-9eff-d42cdb01ad38" (UID: "7246b6cb-1311-42bc-9eff-d42cdb01ad38"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:04.824667 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.824569    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7246b6cb-1311-42bc-9eff-d42cdb01ad38" (UID: "7246b6cb-1311-42bc-9eff-d42cdb01ad38"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:04.826313 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.826294    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7246b6cb-1311-42bc-9eff-d42cdb01ad38-kube-api-access-jl52b" (OuterVolumeSpecName: "kube-api-access-jl52b") pod "7246b6cb-1311-42bc-9eff-d42cdb01ad38" (UID: "7246b6cb-1311-42bc-9eff-d42cdb01ad38"). InnerVolumeSpecName "kube-api-access-jl52b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:05:04.826537 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.826518    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7246b6cb-1311-42bc-9eff-d42cdb01ad38-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7246b6cb-1311-42bc-9eff-d42cdb01ad38" (UID: "7246b6cb-1311-42bc-9eff-d42cdb01ad38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:05:04.826537 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.826524    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-dshm" (OuterVolumeSpecName: "dshm") pod "7246b6cb-1311-42bc-9eff-d42cdb01ad38" (UID: "7246b6cb-1311-42bc-9eff-d42cdb01ad38"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:04.925351 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.925309    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:05:04.925351 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.925343    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7246b6cb-1311-42bc-9eff-d42cdb01ad38-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:05:04.925351 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.925353    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:05:04.925351 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.925360    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:05:04.925351 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.925368    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7246b6cb-1311-42bc-9eff-d42cdb01ad38-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:05:04.925696 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:04.925376    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl52b\" (UniqueName: \"kubernetes.io/projected/7246b6cb-1311-42bc-9eff-d42cdb01ad38-kube-api-access-jl52b\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:05:05.545013 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:05.544979    2569 generic.go:356] "Generic (PLEG): container finished" podID="7246b6cb-1311-42bc-9eff-d42cdb01ad38" containerID="64632728be939bd3a9a4c07624cd96b1522d032f70c99ee8f69e7d709fd1f616" exitCode=0
Jul 29 23:05:05.545178 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:05.545026    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"
Jul 29 23:05:05.545178 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:05.545066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl" event={"ID":"7246b6cb-1311-42bc-9eff-d42cdb01ad38","Type":"ContainerDied","Data":"64632728be939bd3a9a4c07624cd96b1522d032f70c99ee8f69e7d709fd1f616"}
Jul 29 23:05:05.545178 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:05.545127    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl" event={"ID":"7246b6cb-1311-42bc-9eff-d42cdb01ad38","Type":"ContainerDied","Data":"85a0d8a2acac385a812724a47efd861e7c5145ad0d93a8dfa8f6244aa03f5562"}
Jul 29 23:05:05.545178 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:05.545156    2569 scope.go:117] "RemoveContainer" containerID="64632728be939bd3a9a4c07624cd96b1522d032f70c99ee8f69e7d709fd1f616"
Jul 29 23:05:05.553765 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:05.553577    2569 scope.go:117] "RemoveContainer" containerID="64632728be939bd3a9a4c07624cd96b1522d032f70c99ee8f69e7d709fd1f616"
Jul 29 23:05:05.553880 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:05:05.553862    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64632728be939bd3a9a4c07624cd96b1522d032f70c99ee8f69e7d709fd1f616\": container with ID starting with 64632728be939bd3a9a4c07624cd96b1522d032f70c99ee8f69e7d709fd1f616 not found: ID does not exist" containerID="64632728be939bd3a9a4c07624cd96b1522d032f70c99ee8f69e7d709fd1f616"
Jul 29 23:05:05.553925 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:05.553892    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64632728be939bd3a9a4c07624cd96b1522d032f70c99ee8f69e7d709fd1f616"} err="failed to get container status \"64632728be939bd3a9a4c07624cd96b1522d032f70c99ee8f69e7d709fd1f616\": rpc error: code = NotFound desc = could not find container \"64632728be939bd3a9a4c07624cd96b1522d032f70c99ee8f69e7d709fd1f616\": container with ID starting with 64632728be939bd3a9a4c07624cd96b1522d032f70c99ee8f69e7d709fd1f616 not found: ID does not exist"
Jul 29 23:05:05.560538 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:05.560510    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"]
Jul 29 23:05:05.564202 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:05.564179    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9bcjl"]
Jul 29 23:05:07.474009 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:07.473962    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7246b6cb-1311-42bc-9eff-d42cdb01ad38" path="/var/lib/kubelet/pods/7246b6cb-1311-42bc-9eff-d42cdb01ad38/volumes"
Jul 29 23:05:15.177173 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.177117    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"]
Jul 29 23:05:15.177672 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.177412    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7246b6cb-1311-42bc-9eff-d42cdb01ad38" containerName="main"
Jul 29 23:05:15.177672 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.177423    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7246b6cb-1311-42bc-9eff-d42cdb01ad38" containerName="main"
Jul 29 23:05:15.177672 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.177499    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7246b6cb-1311-42bc-9eff-d42cdb01ad38" containerName="main"
Jul 29 23:05:15.181388 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.181359    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.183567 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.183546    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 29 23:05:15.184164 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.184141    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Jul 29 23:05:15.184164 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.184159    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 29 23:05:15.184327 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.184203    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-4wj4x\""
Jul 29 23:05:15.188857 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.188836    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"]
Jul 29 23:05:15.209376 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.209351    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.209517 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.209391    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.209517 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.209450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.209602 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.209518    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.209602 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.209538    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.209602 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.209585    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4scpw\" (UniqueName: \"kubernetes.io/projected/7e97d0ce-e217-4ff6-93a0-217c61561fb1-kube-api-access-4scpw\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.310880 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.310842    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.311062 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.310896    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.311062 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.310935    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.311062 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.310957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.311062 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.311009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4scpw\" (UniqueName: \"kubernetes.io/projected/7e97d0ce-e217-4ff6-93a0-217c61561fb1-kube-api-access-4scpw\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.311263 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.311068    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.311344 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.311322    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.311400 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.311351    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.311507 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.311457    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.313257 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.313229    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.313390 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.313372    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.318745 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.318723    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4scpw\" (UniqueName: \"kubernetes.io/projected/7e97d0ce-e217-4ff6-93a0-217c61561fb1-kube-api-access-4scpw\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.494807 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.494773    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:15.642595 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:15.642563    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"]
Jul 29 23:05:15.646120 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:05:15.646087    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e97d0ce_e217_4ff6_93a0_217c61561fb1.slice/crio-dbf9ca8102c87d825576f1122e2c7b1c066e6fa08cc6e6185233978cdbb103e1 WatchSource:0}: Error finding container dbf9ca8102c87d825576f1122e2c7b1c066e6fa08cc6e6185233978cdbb103e1: Status 404 returned error can't find the container with id dbf9ca8102c87d825576f1122e2c7b1c066e6fa08cc6e6185233978cdbb103e1
Jul 29 23:05:16.589785 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:16.589749    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679" event={"ID":"7e97d0ce-e217-4ff6-93a0-217c61561fb1","Type":"ContainerStarted","Data":"bb1c96bcbbf38a146b5e8c1bd96a21a4a2010ab9e7e06a1a8c26958622b9f5ff"}
Jul 29 23:05:16.590150 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:16.589801    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679" event={"ID":"7e97d0ce-e217-4ff6-93a0-217c61561fb1","Type":"ContainerStarted","Data":"dbf9ca8102c87d825576f1122e2c7b1c066e6fa08cc6e6185233978cdbb103e1"}
Jul 29 23:05:16.606885 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:16.606835    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679" podStartSLOduration=1.606819107 podStartE2EDuration="1.606819107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:05:16.6051306 +0000 UTC m=+1123.610328024" watchObservedRunningTime="2026-07-29 23:05:16.606819107 +0000 UTC m=+1123.612016531"
Jul 29 23:05:17.495666 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:17.495621    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:17.569939 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:17.569900    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"]
Jul 29 23:05:17.632622 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:05:17.632584    2569 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 29 23:05:17.633007 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:05:17.632677    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tls-certs podName:7e97d0ce-e217-4ff6-93a0-217c61561fb1 nodeName:}" failed. No retries permitted until 2026-07-29 23:05:18.132655326 +0000 UTC m=+1125.137852730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679" (UID: "7e97d0ce-e217-4ff6-93a0-217c61561fb1") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 29 23:05:18.137575 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:05:18.137541    2569 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 29 23:05:18.137756 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:05:18.137613    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tls-certs podName:7e97d0ce-e217-4ff6-93a0-217c61561fb1 nodeName:}" failed. No retries permitted until 2026-07-29 23:05:19.137598068 +0000 UTC m=+1126.142795470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679" (UID: "7e97d0ce-e217-4ff6-93a0-217c61561fb1") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 29 23:05:18.609236 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:05:18.609201    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:05:18.609236 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:05:18.609236 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:05:18.609236 ip-10-0-135-98 kubenswrapper[2569]:  > containerID="bb1c96bcbbf38a146b5e8c1bd96a21a4a2010ab9e7e06a1a8c26958622b9f5ff" cmd=["/bin/sleep","15"]
Jul 29 23:05:18.609534 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.609250    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679" message="" err=<
Jul 29 23:05:18.609534 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:05:18.609534 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:05:18.609534 ip-10-0-135-98 kubenswrapper[2569]:  >
Jul 29 23:05:18.609534 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:05:18.609263    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:05:18.609534 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:05:18.609534 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:05:18.609534 ip-10-0-135-98 kubenswrapper[2569]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679" podUID="7e97d0ce-e217-4ff6-93a0-217c61561fb1" containerName="main" containerID="cri-o://bb1c96bcbbf38a146b5e8c1bd96a21a4a2010ab9e7e06a1a8c26958622b9f5ff"
Jul 29 23:05:18.609534 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.609301    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679" podUID="7e97d0ce-e217-4ff6-93a0-217c61561fb1" containerName="main" containerID="cri-o://bb1c96bcbbf38a146b5e8c1bd96a21a4a2010ab9e7e06a1a8c26958622b9f5ff" gracePeriod=60
Jul 29 23:05:18.866919 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.866890    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:18.945106 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.945064    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-home\") pod \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") "
Jul 29 23:05:18.945106 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.945107    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-dshm\") pod \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") "
Jul 29 23:05:18.945321 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.945201    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tls-certs\") pod \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") "
Jul 29 23:05:18.945321 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.945225    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4scpw\" (UniqueName: \"kubernetes.io/projected/7e97d0ce-e217-4ff6-93a0-217c61561fb1-kube-api-access-4scpw\") pod \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") "
Jul 29 23:05:18.945321 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.945256    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tmp-dir\") pod \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") "
Jul 29 23:05:18.945321 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.945284    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-model-cache\") pod \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\" (UID: \"7e97d0ce-e217-4ff6-93a0-217c61561fb1\") "
Jul 29 23:05:18.945550 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.945340    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-home" (OuterVolumeSpecName: "home") pod "7e97d0ce-e217-4ff6-93a0-217c61561fb1" (UID: "7e97d0ce-e217-4ff6-93a0-217c61561fb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:18.945550 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.945539    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:05:18.945655 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.945543    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e97d0ce-e217-4ff6-93a0-217c61561fb1" (UID: "7e97d0ce-e217-4ff6-93a0-217c61561fb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:18.945655 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.945623    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e97d0ce-e217-4ff6-93a0-217c61561fb1" (UID: "7e97d0ce-e217-4ff6-93a0-217c61561fb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:18.947468 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.947436    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e97d0ce-e217-4ff6-93a0-217c61561fb1-kube-api-access-4scpw" (OuterVolumeSpecName: "kube-api-access-4scpw") pod "7e97d0ce-e217-4ff6-93a0-217c61561fb1" (UID: "7e97d0ce-e217-4ff6-93a0-217c61561fb1"). InnerVolumeSpecName "kube-api-access-4scpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:05:18.947468 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.947459    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-dshm" (OuterVolumeSpecName: "dshm") pod "7e97d0ce-e217-4ff6-93a0-217c61561fb1" (UID: "7e97d0ce-e217-4ff6-93a0-217c61561fb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:18.947468 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:18.947458    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e97d0ce-e217-4ff6-93a0-217c61561fb1" (UID: "7e97d0ce-e217-4ff6-93a0-217c61561fb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:05:19.046197 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:19.046154    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:05:19.046197 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:19.046191    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:05:19.046197 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:19.046202    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4scpw\" (UniqueName: \"kubernetes.io/projected/7e97d0ce-e217-4ff6-93a0-217c61561fb1-kube-api-access-4scpw\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:05:19.046441 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:19.046212    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:05:19.046441 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:19.046220    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e97d0ce-e217-4ff6-93a0-217c61561fb1-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:05:19.603969 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:19.603933    2569 generic.go:356] "Generic (PLEG): container finished" podID="7e97d0ce-e217-4ff6-93a0-217c61561fb1" containerID="bb1c96bcbbf38a146b5e8c1bd96a21a4a2010ab9e7e06a1a8c26958622b9f5ff" exitCode=0
Jul 29 23:05:19.604136 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:19.603985    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"
Jul 29 23:05:19.604136 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:19.604009    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679" event={"ID":"7e97d0ce-e217-4ff6-93a0-217c61561fb1","Type":"ContainerDied","Data":"bb1c96bcbbf38a146b5e8c1bd96a21a4a2010ab9e7e06a1a8c26958622b9f5ff"}
Jul 29 23:05:19.604136 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:19.604060    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679" event={"ID":"7e97d0ce-e217-4ff6-93a0-217c61561fb1","Type":"ContainerDied","Data":"dbf9ca8102c87d825576f1122e2c7b1c066e6fa08cc6e6185233978cdbb103e1"}
Jul 29 23:05:19.604136 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:19.604090    2569 scope.go:117] "RemoveContainer" containerID="bb1c96bcbbf38a146b5e8c1bd96a21a4a2010ab9e7e06a1a8c26958622b9f5ff"
Jul 29 23:05:19.614261 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:19.614090    2569 scope.go:117] "RemoveContainer" containerID="bb1c96bcbbf38a146b5e8c1bd96a21a4a2010ab9e7e06a1a8c26958622b9f5ff"
Jul 29 23:05:19.614528 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:05:19.614506    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb1c96bcbbf38a146b5e8c1bd96a21a4a2010ab9e7e06a1a8c26958622b9f5ff\": container with ID starting with bb1c96bcbbf38a146b5e8c1bd96a21a4a2010ab9e7e06a1a8c26958622b9f5ff not found: ID does not exist" containerID="bb1c96bcbbf38a146b5e8c1bd96a21a4a2010ab9e7e06a1a8c26958622b9f5ff"
Jul 29 23:05:19.614588 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:19.614536    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1c96bcbbf38a146b5e8c1bd96a21a4a2010ab9e7e06a1a8c26958622b9f5ff"} err="failed to get container status \"bb1c96bcbbf38a146b5e8c1bd96a21a4a2010ab9e7e06a1a8c26958622b9f5ff\": rpc error: code = NotFound desc = could not find container \"bb1c96bcbbf38a146b5e8c1bd96a21a4a2010ab9e7e06a1a8c26958622b9f5ff\": container with ID starting with bb1c96bcbbf38a146b5e8c1bd96a21a4a2010ab9e7e06a1a8c26958622b9f5ff not found: ID does not exist"
Jul 29 23:05:19.618642 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:19.618618    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"]
Jul 29 23:05:19.622548 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:19.622524    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdz679"]
Jul 29 23:05:21.473625 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:21.473592    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e97d0ce-e217-4ff6-93a0-217c61561fb1" path="/var/lib/kubelet/pods/7e97d0ce-e217-4ff6-93a0-217c61561fb1/volumes"
Jul 29 23:05:28.005661 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.005623    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"]
Jul 29 23:05:28.006166 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.006023    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e97d0ce-e217-4ff6-93a0-217c61561fb1" containerName="main"
Jul 29 23:05:28.006166 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.006040    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e97d0ce-e217-4ff6-93a0-217c61561fb1" containerName="main"
Jul 29 23:05:28.006166 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.006100    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e97d0ce-e217-4ff6-93a0-217c61561fb1" containerName="main"
Jul 29 23:05:28.010329 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.010306    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.012545 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.012517    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Jul 29 23:05:28.012545 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.012531    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Jul 29 23:05:28.012545 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.012532    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 29 23:05:28.012861 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.012612    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Jul 29 23:05:28.012861 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.012622    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-k225s\""
Jul 29 23:05:28.019453 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.019432    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"]
Jul 29 23:05:28.123663 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.123628    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-dshm\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.123663 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.123665    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-tmp-dir\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.123881 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.123683    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r8xh\" (UniqueName: \"kubernetes.io/projected/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-kube-api-access-8r8xh\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.123881 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.123702    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-kserve-provision-location\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.123881 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.123757    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-tls-certs\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.123881 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.123812    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-home\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.123881 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.123853    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-model-cache\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.225095 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.225058    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-home\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.225267 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.225115    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-model-cache\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.225267 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.225156    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-dshm\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.225267 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.225172    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-tmp-dir\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.225267 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.225189    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r8xh\" (UniqueName: \"kubernetes.io/projected/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-kube-api-access-8r8xh\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.225267 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.225207    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-kserve-provision-location\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.225525 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.225287    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-tls-certs\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.225605 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.225543    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-home\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.225670 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.225615    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-tmp-dir\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.225670 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.225645    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-model-cache\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.225773 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.225685    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-kserve-provision-location\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.227334 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.227305    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-dshm\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.227848 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.227823    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-tls-certs\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.233435 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.233413    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r8xh\" (UniqueName: \"kubernetes.io/projected/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-kube-api-access-8r8xh\") pod \"auth-enabled-test-kserve-6979c7746b-6vsj7\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.322461 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.322368    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:05:28.466365 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.466335    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"]
Jul 29 23:05:28.469499 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:05:28.469455    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e20d93e_005c_4e84_9e31_8ffcfe5727b9.slice/crio-cefe2aeddf66d6e4433c5e942a17ec8ad366bf538eec5792c616b977ca488fb1 WatchSource:0}: Error finding container cefe2aeddf66d6e4433c5e942a17ec8ad366bf538eec5792c616b977ca488fb1: Status 404 returned error can't find the container with id cefe2aeddf66d6e4433c5e942a17ec8ad366bf538eec5792c616b977ca488fb1
Jul 29 23:05:28.642385 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.642295    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" event={"ID":"5e20d93e-005c-4e84-9e31-8ffcfe5727b9","Type":"ContainerStarted","Data":"7c5e5cab58c0eeaace5790956073457439b928d4ce9bd2d189963f4c47c224aa"}
Jul 29 23:05:28.642385 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:28.642346    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" event={"ID":"5e20d93e-005c-4e84-9e31-8ffcfe5727b9","Type":"ContainerStarted","Data":"cefe2aeddf66d6e4433c5e942a17ec8ad366bf538eec5792c616b977ca488fb1"}
Jul 29 23:05:33.663658 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:33.663572    2569 generic.go:356] "Generic (PLEG): container finished" podID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerID="7c5e5cab58c0eeaace5790956073457439b928d4ce9bd2d189963f4c47c224aa" exitCode=0
Jul 29 23:05:33.664105 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:05:33.663644    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" event={"ID":"5e20d93e-005c-4e84-9e31-8ffcfe5727b9","Type":"ContainerDied","Data":"7c5e5cab58c0eeaace5790956073457439b928d4ce9bd2d189963f4c47c224aa"}
Jul 29 23:06:02.809160 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:02.809119    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" event={"ID":"5e20d93e-005c-4e84-9e31-8ffcfe5727b9","Type":"ContainerStarted","Data":"0e120b400bb90e070782b18c6c43be0191de2cebf7bf0330dc1684b7181a14f2"}
Jul 29 23:06:02.830616 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:02.830565    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podStartSLOduration=2.243624602 podStartE2EDuration="35.830548768s" totalImagesPullingTime="28.640230185s" totalInitContainerRuntime="4.946693981s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:05:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:06:02.829742424 +0000 UTC m=+1169.834939859" watchObservedRunningTime="2026-07-29 23:06:02.830548768 +0000 UTC m=+1169.835746190"
Jul 29 23:06:03.168604 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.168515    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 29 23:06:03.170798 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.170774    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.174908 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.174885    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 29 23:06:03.175042 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.175010    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 29 23:06:03.175123 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.175070    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-lhrnv\""
Jul 29 23:06:03.175203 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.175182    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-h8vhb\""
Jul 29 23:06:03.175444 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.175420    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 29 23:06:03.175627 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.175610    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 29 23:06:03.185858 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.185836    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 29 23:06:03.258544 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.258505    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxr8v\" (UniqueName: \"kubernetes.io/projected/fdd9bd09-2791-46cf-a14e-ea6229e50fda-kube-api-access-cxr8v\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.258722 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.258550    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.258722 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.258613    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.258722 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.258639    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.258840 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.258727    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.258840 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.258760    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.258840 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.258823    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.323046 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.323011    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:06:03.359793 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.359747    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.359793 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.359794    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.360048 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.359837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.360048 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.359874    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxr8v\" (UniqueName: \"kubernetes.io/projected/fdd9bd09-2791-46cf-a14e-ea6229e50fda-kube-api-access-cxr8v\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.360048 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.359899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.360048 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.359930    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.360246 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.360197    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.360299 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.360244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.360367 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.360253    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.360440 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.360423    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.360511 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.360439    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.362158 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.362135    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.362322 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.362307    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.369167 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.369142    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxr8v\" (UniqueName: \"kubernetes.io/projected/fdd9bd09-2791-46cf-a14e-ea6229e50fda-kube-api-access-cxr8v\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.481564 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.481533    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:06:03.622940 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.622906    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 29 23:06:03.626149 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:06:03.626118    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd9bd09_2791_46cf_a14e_ea6229e50fda.slice/crio-41ba791cbb6aaa30dcd67c2b2ae98901097f5289b7e3b39226f650f96b2f2dd7 WatchSource:0}: Error finding container 41ba791cbb6aaa30dcd67c2b2ae98901097f5289b7e3b39226f650f96b2f2dd7: Status 404 returned error can't find the container with id 41ba791cbb6aaa30dcd67c2b2ae98901097f5289b7e3b39226f650f96b2f2dd7
Jul 29 23:06:03.815677 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.815585    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fdd9bd09-2791-46cf-a14e-ea6229e50fda","Type":"ContainerStarted","Data":"a7d9c360a70c3fbf0e64c24e817be0bc6e096f24a16a015e0428ca19165dcacb"}
Jul 29 23:06:03.815677 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:03.815650    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fdd9bd09-2791-46cf-a14e-ea6229e50fda","Type":"ContainerStarted","Data":"41ba791cbb6aaa30dcd67c2b2ae98901097f5289b7e3b39226f650f96b2f2dd7"}
Jul 29 23:06:08.323521 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:08.323455    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:06:08.325207 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:08.325174    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 23:06:10.848929 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:10.848889    2569 generic.go:356] "Generic (PLEG): container finished" podID="fdd9bd09-2791-46cf-a14e-ea6229e50fda" containerID="a7d9c360a70c3fbf0e64c24e817be0bc6e096f24a16a015e0428ca19165dcacb" exitCode=0
Jul 29 23:06:10.849393 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:10.848952    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fdd9bd09-2791-46cf-a14e-ea6229e50fda","Type":"ContainerDied","Data":"a7d9c360a70c3fbf0e64c24e817be0bc6e096f24a16a015e0428ca19165dcacb"}
Jul 29 23:06:11.855602 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:11.855562    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fdd9bd09-2791-46cf-a14e-ea6229e50fda","Type":"ContainerStarted","Data":"abef802ad21d07ae9ef54c01b4c06fbac9240581f513a8932472985aeba4325a"}
Jul 29 23:06:11.877598 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:11.877521    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.3452377110000002 podStartE2EDuration="8.877431532s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.532193821s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:06:11.877138968 +0000 UTC m=+1178.882336404" watchObservedRunningTime="2026-07-29 23:06:11.877431532 +0000 UTC m=+1178.882628957"
Jul 29 23:06:18.323564 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:18.323507    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 23:06:28.323722 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:28.323664    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 23:06:33.493043 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:33.493014    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:06:33.495463 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:33.495437    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:06:38.323869 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:38.323823    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 23:06:48.323365 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:48.323310    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 23:06:58.323451 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:06:58.323398    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 23:07:08.323886 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:07:08.323841    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 23:07:18.322993 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:07:18.322950    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 23:07:28.323041 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:07:28.322933    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 23:07:38.323366 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:07:38.323324    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 23:07:48.322953 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:07:48.322909    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 23:07:58.323044 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:07:58.322992    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 23:08:08.323478 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:08.323426    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 23:08:18.323563 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:18.323521    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 23:08:28.334141 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:28.334110    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:08:28.342118 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:28.342083    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:08:33.369938 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:33.369901    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"]
Jul 29 23:08:48.398580 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:48.398520    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main" containerID="cri-o://0e120b400bb90e070782b18c6c43be0191de2cebf7bf0330dc1684b7181a14f2" gracePeriod=45
Jul 29 23:08:49.143202 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.143177    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:08:49.234471 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.234441    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-home\") pod \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") "
Jul 29 23:08:49.234652 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.234510    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-model-cache\") pod \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") "
Jul 29 23:08:49.234652 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.234541    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-dshm\") pod \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") "
Jul 29 23:08:49.234652 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.234581    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-tmp-dir\") pod \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") "
Jul 29 23:08:49.234652 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.234607    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-tls-certs\") pod \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") "
Jul 29 23:08:49.234652 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.234640    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-kserve-provision-location\") pod \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") "
Jul 29 23:08:49.234936 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.234671    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r8xh\" (UniqueName: \"kubernetes.io/projected/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-kube-api-access-8r8xh\") pod \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\" (UID: \"5e20d93e-005c-4e84-9e31-8ffcfe5727b9\") "
Jul 29 23:08:49.234936 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.234788    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e20d93e-005c-4e84-9e31-8ffcfe5727b9" (UID: "5e20d93e-005c-4e84-9e31-8ffcfe5727b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:08:49.235054 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.234976    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:08:49.235240 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.235205    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-home" (OuterVolumeSpecName: "home") pod "5e20d93e-005c-4e84-9e31-8ffcfe5727b9" (UID: "5e20d93e-005c-4e84-9e31-8ffcfe5727b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:08:49.237352 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.237304    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-dshm" (OuterVolumeSpecName: "dshm") pod "5e20d93e-005c-4e84-9e31-8ffcfe5727b9" (UID: "5e20d93e-005c-4e84-9e31-8ffcfe5727b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:08:49.237445 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.237381    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e20d93e-005c-4e84-9e31-8ffcfe5727b9" (UID: "5e20d93e-005c-4e84-9e31-8ffcfe5727b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:08:49.237445 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.237423    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-kube-api-access-8r8xh" (OuterVolumeSpecName: "kube-api-access-8r8xh") pod "5e20d93e-005c-4e84-9e31-8ffcfe5727b9" (UID: "5e20d93e-005c-4e84-9e31-8ffcfe5727b9"). InnerVolumeSpecName "kube-api-access-8r8xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:08:49.255037 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.254993    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e20d93e-005c-4e84-9e31-8ffcfe5727b9" (UID: "5e20d93e-005c-4e84-9e31-8ffcfe5727b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:08:49.292122 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.292062    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e20d93e-005c-4e84-9e31-8ffcfe5727b9" (UID: "5e20d93e-005c-4e84-9e31-8ffcfe5727b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:08:49.335792 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.335759    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-kserve-provision-location\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:08:49.335792 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.335789    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r8xh\" (UniqueName: \"kubernetes.io/projected/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-kube-api-access-8r8xh\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:08:49.335972 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.335800    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:08:49.335972 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.335809    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:08:49.335972 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.335835    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:08:49.335972 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.335843    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e20d93e-005c-4e84-9e31-8ffcfe5727b9-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:08:49.559061 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.558962    2569 generic.go:356] "Generic (PLEG): container finished" podID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerID="0e120b400bb90e070782b18c6c43be0191de2cebf7bf0330dc1684b7181a14f2" exitCode=0
Jul 29 23:08:49.559061 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.559021    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"
Jul 29 23:08:49.559061 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.559044    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" event={"ID":"5e20d93e-005c-4e84-9e31-8ffcfe5727b9","Type":"ContainerDied","Data":"0e120b400bb90e070782b18c6c43be0191de2cebf7bf0330dc1684b7181a14f2"}
Jul 29 23:08:49.559560 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.559095    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7" event={"ID":"5e20d93e-005c-4e84-9e31-8ffcfe5727b9","Type":"ContainerDied","Data":"cefe2aeddf66d6e4433c5e942a17ec8ad366bf538eec5792c616b977ca488fb1"}
Jul 29 23:08:49.559560 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.559120    2569 scope.go:117] "RemoveContainer" containerID="0e120b400bb90e070782b18c6c43be0191de2cebf7bf0330dc1684b7181a14f2"
Jul 29 23:08:49.567327 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.567307    2569 scope.go:117] "RemoveContainer" containerID="7c5e5cab58c0eeaace5790956073457439b928d4ce9bd2d189963f4c47c224aa"
Jul 29 23:08:49.575405 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.575290    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"]
Jul 29 23:08:49.575493 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.575422    2569 scope.go:117] "RemoveContainer" containerID="0e120b400bb90e070782b18c6c43be0191de2cebf7bf0330dc1684b7181a14f2"
Jul 29 23:08:49.575703 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:08:49.575684    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e120b400bb90e070782b18c6c43be0191de2cebf7bf0330dc1684b7181a14f2\": container with ID starting with 0e120b400bb90e070782b18c6c43be0191de2cebf7bf0330dc1684b7181a14f2 not found: ID does not exist" containerID="0e120b400bb90e070782b18c6c43be0191de2cebf7bf0330dc1684b7181a14f2"
Jul 29 23:08:49.575780 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.575710    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e120b400bb90e070782b18c6c43be0191de2cebf7bf0330dc1684b7181a14f2"} err="failed to get container status \"0e120b400bb90e070782b18c6c43be0191de2cebf7bf0330dc1684b7181a14f2\": rpc error: code = NotFound desc = could not find container \"0e120b400bb90e070782b18c6c43be0191de2cebf7bf0330dc1684b7181a14f2\": container with ID starting with 0e120b400bb90e070782b18c6c43be0191de2cebf7bf0330dc1684b7181a14f2 not found: ID does not exist"
Jul 29 23:08:49.575780 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.575727    2569 scope.go:117] "RemoveContainer" containerID="7c5e5cab58c0eeaace5790956073457439b928d4ce9bd2d189963f4c47c224aa"
Jul 29 23:08:49.575967 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:08:49.575949    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5e5cab58c0eeaace5790956073457439b928d4ce9bd2d189963f4c47c224aa\": container with ID starting with 7c5e5cab58c0eeaace5790956073457439b928d4ce9bd2d189963f4c47c224aa not found: ID does not exist" containerID="7c5e5cab58c0eeaace5790956073457439b928d4ce9bd2d189963f4c47c224aa"
Jul 29 23:08:49.576016 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.575978    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5e5cab58c0eeaace5790956073457439b928d4ce9bd2d189963f4c47c224aa"} err="failed to get container status \"7c5e5cab58c0eeaace5790956073457439b928d4ce9bd2d189963f4c47c224aa\": rpc error: code = NotFound desc = could not find container \"7c5e5cab58c0eeaace5790956073457439b928d4ce9bd2d189963f4c47c224aa\": container with ID starting with 7c5e5cab58c0eeaace5790956073457439b928d4ce9bd2d189963f4c47c224aa not found: ID does not exist"
Jul 29 23:08:49.580262 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:49.580239    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6979c7746b-6vsj7"]
Jul 29 23:08:51.469639 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:51.469600    2569 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-h8vhb\" not found"
Jul 29 23:08:51.474356 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:51.474327    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" path="/var/lib/kubelet/pods/5e20d93e-005c-4e84-9e31-8ffcfe5727b9/volumes"
Jul 29 23:08:51.552550 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:08:51.552508    2569 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 23:08:51.552720 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:08:51.552600    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs podName:fdd9bd09-2791-46cf-a14e-ea6229e50fda nodeName:}" failed. No retries permitted until 2026-07-29 23:08:52.052579409 +0000 UTC m=+1339.057776811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "fdd9bd09-2791-46cf-a14e-ea6229e50fda") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 23:08:52.055936 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:08:52.055884    2569 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 23:08:52.056123 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:08:52.055968    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs podName:fdd9bd09-2791-46cf-a14e-ea6229e50fda nodeName:}" failed. No retries permitted until 2026-07-29 23:08:53.055954589 +0000 UTC m=+1340.061151991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "fdd9bd09-2791-46cf-a14e-ea6229e50fda") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 23:08:53.062247 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:08:53.062210    2569 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 23:08:53.062654 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:08:53.062281    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs podName:fdd9bd09-2791-46cf-a14e-ea6229e50fda nodeName:}" failed. No retries permitted until 2026-07-29 23:08:55.06226568 +0000 UTC m=+1342.067463081 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "fdd9bd09-2791-46cf-a14e-ea6229e50fda") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 23:08:54.907282 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:08:54.907244    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 29 23:08:55.078114 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:08:55.078078    2569 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 23:08:55.078283 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:08:55.078151    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs podName:fdd9bd09-2791-46cf-a14e-ea6229e50fda nodeName:}" failed. No retries permitted until 2026-07-29 23:08:59.078135452 +0000 UTC m=+1346.083332861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "fdd9bd09-2791-46cf-a14e-ea6229e50fda") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 23:08:59.112390 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:08:59.112342    2569 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 23:08:59.112843 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:08:59.112425    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs podName:fdd9bd09-2791-46cf-a14e-ea6229e50fda nodeName:}" failed. No retries permitted until 2026-07-29 23:09:07.112410318 +0000 UTC m=+1354.117607718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "fdd9bd09-2791-46cf-a14e-ea6229e50fda") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 23:09:07.181960 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:09:07.181924    2569 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 23:09:07.182534 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:09:07.181999    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs podName:fdd9bd09-2791-46cf-a14e-ea6229e50fda nodeName:}" failed. No retries permitted until 2026-07-29 23:09:23.18198285 +0000 UTC m=+1370.187180251 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "fdd9bd09-2791-46cf-a14e-ea6229e50fda") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 23:09:09.928781 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:09.928739    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="fdd9bd09-2791-46cf-a14e-ea6229e50fda" containerName="main" containerID="cri-o://abef802ad21d07ae9ef54c01b4c06fbac9240581f513a8932472985aeba4325a" gracePeriod=45
Jul 29 23:09:10.597160 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.597127    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:09:10.650259 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.650223    2569 generic.go:356] "Generic (PLEG): container finished" podID="fdd9bd09-2791-46cf-a14e-ea6229e50fda" containerID="abef802ad21d07ae9ef54c01b4c06fbac9240581f513a8932472985aeba4325a" exitCode=0
Jul 29 23:09:10.650259 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.650268    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fdd9bd09-2791-46cf-a14e-ea6229e50fda","Type":"ContainerDied","Data":"abef802ad21d07ae9ef54c01b4c06fbac9240581f513a8932472985aeba4325a"}
Jul 29 23:09:10.650473 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.650283    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 23:09:10.650473 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.650360    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fdd9bd09-2791-46cf-a14e-ea6229e50fda","Type":"ContainerDied","Data":"41ba791cbb6aaa30dcd67c2b2ae98901097f5289b7e3b39226f650f96b2f2dd7"}
Jul 29 23:09:10.650473 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.650392    2569 scope.go:117] "RemoveContainer" containerID="abef802ad21d07ae9ef54c01b4c06fbac9240581f513a8932472985aeba4325a"
Jul 29 23:09:10.658375 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.658352    2569 scope.go:117] "RemoveContainer" containerID="a7d9c360a70c3fbf0e64c24e817be0bc6e096f24a16a015e0428ca19165dcacb"
Jul 29 23:09:10.665957 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.665935    2569 scope.go:117] "RemoveContainer" containerID="abef802ad21d07ae9ef54c01b4c06fbac9240581f513a8932472985aeba4325a"
Jul 29 23:09:10.666224 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:09:10.666207    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abef802ad21d07ae9ef54c01b4c06fbac9240581f513a8932472985aeba4325a\": container with ID starting with abef802ad21d07ae9ef54c01b4c06fbac9240581f513a8932472985aeba4325a not found: ID does not exist" containerID="abef802ad21d07ae9ef54c01b4c06fbac9240581f513a8932472985aeba4325a"
Jul 29 23:09:10.666269 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.666234    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abef802ad21d07ae9ef54c01b4c06fbac9240581f513a8932472985aeba4325a"} err="failed to get container status \"abef802ad21d07ae9ef54c01b4c06fbac9240581f513a8932472985aeba4325a\": rpc error: code = NotFound desc = could not find container \"abef802ad21d07ae9ef54c01b4c06fbac9240581f513a8932472985aeba4325a\": container with ID starting with abef802ad21d07ae9ef54c01b4c06fbac9240581f513a8932472985aeba4325a not found: ID does not exist"
Jul 29 23:09:10.666269 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.666252    2569 scope.go:117] "RemoveContainer" containerID="a7d9c360a70c3fbf0e64c24e817be0bc6e096f24a16a015e0428ca19165dcacb"
Jul 29 23:09:10.666515 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:09:10.666475    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d9c360a70c3fbf0e64c24e817be0bc6e096f24a16a015e0428ca19165dcacb\": container with ID starting with a7d9c360a70c3fbf0e64c24e817be0bc6e096f24a16a015e0428ca19165dcacb not found: ID does not exist" containerID="a7d9c360a70c3fbf0e64c24e817be0bc6e096f24a16a015e0428ca19165dcacb"
Jul 29 23:09:10.666578 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.666524    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d9c360a70c3fbf0e64c24e817be0bc6e096f24a16a015e0428ca19165dcacb"} err="failed to get container status \"a7d9c360a70c3fbf0e64c24e817be0bc6e096f24a16a015e0428ca19165dcacb\": rpc error: code = NotFound desc = could not find container \"a7d9c360a70c3fbf0e64c24e817be0bc6e096f24a16a015e0428ca19165dcacb\": container with ID starting with a7d9c360a70c3fbf0e64c24e817be0bc6e096f24a16a015e0428ca19165dcacb not found: ID does not exist"
Jul 29 23:09:10.712006 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.711928    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-kserve-provision-location\") pod \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") "
Jul 29 23:09:10.712006 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.711969    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs\") pod \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") "
Jul 29 23:09:10.712150 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.712015    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tmp-dir\") pod \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") "
Jul 29 23:09:10.712150 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.712037    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-home\") pod \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") "
Jul 29 23:09:10.712150 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.712062    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxr8v\" (UniqueName: \"kubernetes.io/projected/fdd9bd09-2791-46cf-a14e-ea6229e50fda-kube-api-access-cxr8v\") pod \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") "
Jul 29 23:09:10.712150 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.712121    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-dshm\") pod \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") "
Jul 29 23:09:10.712349 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.712159    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-model-cache\") pod \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\" (UID: \"fdd9bd09-2791-46cf-a14e-ea6229e50fda\") "
Jul 29 23:09:10.712711 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.712664    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdd9bd09-2791-46cf-a14e-ea6229e50fda" (UID: "fdd9bd09-2791-46cf-a14e-ea6229e50fda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:09:10.712896 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.712857    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-home" (OuterVolumeSpecName: "home") pod "fdd9bd09-2791-46cf-a14e-ea6229e50fda" (UID: "fdd9bd09-2791-46cf-a14e-ea6229e50fda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:09:10.714426 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.714399    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdd9bd09-2791-46cf-a14e-ea6229e50fda" (UID: "fdd9bd09-2791-46cf-a14e-ea6229e50fda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:09:10.714555 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.714454    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd9bd09-2791-46cf-a14e-ea6229e50fda-kube-api-access-cxr8v" (OuterVolumeSpecName: "kube-api-access-cxr8v") pod "fdd9bd09-2791-46cf-a14e-ea6229e50fda" (UID: "fdd9bd09-2791-46cf-a14e-ea6229e50fda"). InnerVolumeSpecName "kube-api-access-cxr8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:09:10.714555 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.714523    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-dshm" (OuterVolumeSpecName: "dshm") pod "fdd9bd09-2791-46cf-a14e-ea6229e50fda" (UID: "fdd9bd09-2791-46cf-a14e-ea6229e50fda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:09:10.725062 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.725036    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdd9bd09-2791-46cf-a14e-ea6229e50fda" (UID: "fdd9bd09-2791-46cf-a14e-ea6229e50fda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:09:10.769935 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.769892    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fdd9bd09-2791-46cf-a14e-ea6229e50fda" (UID: "fdd9bd09-2791-46cf-a14e-ea6229e50fda"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:09:10.812827 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.812791    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxr8v\" (UniqueName: \"kubernetes.io/projected/fdd9bd09-2791-46cf-a14e-ea6229e50fda-kube-api-access-cxr8v\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:09:10.812827 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.812819    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:09:10.812827 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.812829    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:09:10.813042 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.812839    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-kserve-provision-location\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:09:10.813042 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.812847    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:09:10.813042 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.812857    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:09:10.813042 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.812864    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd9bd09-2791-46cf-a14e-ea6229e50fda-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:09:10.973861 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.973818    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 29 23:09:10.979838 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:10.979812    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 29 23:09:11.475109 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:11.475074    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdd9bd09-2791-46cf-a14e-ea6229e50fda" path="/var/lib/kubelet/pods/fdd9bd09-2791-46cf-a14e-ea6229e50fda/volumes"
Jul 29 23:09:19.889351 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.889313    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"]
Jul 29 23:09:19.889843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.889619    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main"
Jul 29 23:09:19.889843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.889631    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main"
Jul 29 23:09:19.889843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.889644    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd9bd09-2791-46cf-a14e-ea6229e50fda" containerName="main"
Jul 29 23:09:19.889843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.889650    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd9bd09-2791-46cf-a14e-ea6229e50fda" containerName="main"
Jul 29 23:09:19.889843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.889658    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="storage-initializer"
Jul 29 23:09:19.889843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.889664    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="storage-initializer"
Jul 29 23:09:19.889843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.889673    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd9bd09-2791-46cf-a14e-ea6229e50fda" containerName="storage-initializer"
Jul 29 23:09:19.889843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.889679    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd9bd09-2791-46cf-a14e-ea6229e50fda" containerName="storage-initializer"
Jul 29 23:09:19.889843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.889736    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e20d93e-005c-4e84-9e31-8ffcfe5727b9" containerName="main"
Jul 29 23:09:19.889843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.889744    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd9bd09-2791-46cf-a14e-ea6229e50fda" containerName="main"
Jul 29 23:09:19.894279 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.894258    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:19.896306 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.896281    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 29 23:09:19.896768 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.896738    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 29 23:09:19.896876 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.896791    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-jzxzj\""
Jul 29 23:09:19.896876 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.896834    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 29 23:09:19.901742 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.901587    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"]
Jul 29 23:09:19.991938 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.991897    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:19.991938 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.991940    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:19.992163 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.991974    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:19.992163 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.992014    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:19.992163 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.992043    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:19.992163 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:19.992065    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnzdk\" (UniqueName: \"kubernetes.io/projected/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-kube-api-access-lnzdk\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:20.093363 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.093324    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:20.093571 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.093374    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:20.093571 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.093519    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnzdk\" (UniqueName: \"kubernetes.io/projected/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-kube-api-access-lnzdk\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:20.093666 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.093592    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:20.093666 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.093619    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:20.093766 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.093667    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:20.094030 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.093995    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:20.094163 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.094029    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:20.094163 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.094053    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:20.095997 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.095970    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:20.096455 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.096429    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:20.104302 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.104268    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnzdk\" (UniqueName: \"kubernetes.io/projected/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-kube-api-access-lnzdk\") pod \"scheduler-inline-config-test-kserve-8655bc757-g6n88\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:20.207631 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.207531    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:20.339962 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.339932    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"]
Jul 29 23:09:20.342741 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:09:20.342715    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cdb517f_bf5b_49a6_a4d1_4c7e4c1a3cb4.slice/crio-f04144b0756dd98ecbe49a72fc5146d7a99598d5fa5c0afc646c60316e331d8b WatchSource:0}: Error finding container f04144b0756dd98ecbe49a72fc5146d7a99598d5fa5c0afc646c60316e331d8b: Status 404 returned error can't find the container with id f04144b0756dd98ecbe49a72fc5146d7a99598d5fa5c0afc646c60316e331d8b
Jul 29 23:09:20.692907 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.692868    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88" event={"ID":"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4","Type":"ContainerStarted","Data":"ff91d2cac25745ac783eaede38047ae6cea06c7025c2c62c23aee0e54e30ab25"}
Jul 29 23:09:20.693092 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.692924    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88" event={"ID":"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4","Type":"ContainerStarted","Data":"f04144b0756dd98ecbe49a72fc5146d7a99598d5fa5c0afc646c60316e331d8b"}
Jul 29 23:09:20.713558 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:20.713505    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88" podStartSLOduration=1.713470552 podStartE2EDuration="1.713470552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:09:20.711923277 +0000 UTC m=+1367.717120724" watchObservedRunningTime="2026-07-29 23:09:20.713470552 +0000 UTC m=+1367.718668010"
Jul 29 23:09:21.208313 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:21.208274    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:30.208607 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:30.208567    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:30.220926 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:30.220899    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:30.746633 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:30.746600    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:52.802372 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:52.802337    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"]
Jul 29 23:09:52.811172 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:09:52.811131    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:09:52.811172 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:09:52.811172 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:09:52.811172 ip-10-0-135-98 kubenswrapper[2569]:  > containerID="ff91d2cac25745ac783eaede38047ae6cea06c7025c2c62c23aee0e54e30ab25" cmd=["/bin/sleep","15"]
Jul 29 23:09:52.811444 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:52.811188    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88" message="" err=<
Jul 29 23:09:52.811444 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:09:52.811444 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:09:52.811444 ip-10-0-135-98 kubenswrapper[2569]:  >
Jul 29 23:09:52.811444 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:09:52.811201    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:09:52.811444 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:09:52.811444 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:09:52.811444 ip-10-0-135-98 kubenswrapper[2569]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88" podUID="4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4" containerName="main" containerID="cri-o://ff91d2cac25745ac783eaede38047ae6cea06c7025c2c62c23aee0e54e30ab25"
Jul 29 23:09:52.811444 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:52.811240    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88" podUID="4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4" containerName="main" containerID="cri-o://ff91d2cac25745ac783eaede38047ae6cea06c7025c2c62c23aee0e54e30ab25" gracePeriod=60
Jul 29 23:09:53.557575 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.557545    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:53.576736 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.576700    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-home\") pod \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") "
Jul 29 23:09:53.576895 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.576760    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-dshm\") pod \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") "
Jul 29 23:09:53.576895 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.576802    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-model-cache\") pod \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") "
Jul 29 23:09:53.576895 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.576831    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-tls-certs\") pod \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") "
Jul 29 23:09:53.577074 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.576911    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-tmp-dir\") pod \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") "
Jul 29 23:09:53.577074 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.576938    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnzdk\" (UniqueName: \"kubernetes.io/projected/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-kube-api-access-lnzdk\") pod \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\" (UID: \"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4\") "
Jul 29 23:09:53.577074 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.576973    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-home" (OuterVolumeSpecName: "home") pod "4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4" (UID: "4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:09:53.577222 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.577069    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-model-cache" (OuterVolumeSpecName: "model-cache") pod "4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4" (UID: "4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:09:53.577222 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.577185    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4" (UID: "4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:09:53.577420 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.577388    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:09:53.577420 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.577409    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:09:53.577420 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.577421    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:09:53.579475 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.579444    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-dshm" (OuterVolumeSpecName: "dshm") pod "4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4" (UID: "4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:09:53.579643 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.579618    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4" (UID: "4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:09:53.579795 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.579770    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-kube-api-access-lnzdk" (OuterVolumeSpecName: "kube-api-access-lnzdk") pod "4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4" (UID: "4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4"). InnerVolumeSpecName "kube-api-access-lnzdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:09:53.678151 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.678050    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnzdk\" (UniqueName: \"kubernetes.io/projected/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-kube-api-access-lnzdk\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:09:53.678151 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.678089    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:09:53.678151 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.678101    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:09:53.823666 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.823629    2569 generic.go:356] "Generic (PLEG): container finished" podID="4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4" containerID="ff91d2cac25745ac783eaede38047ae6cea06c7025c2c62c23aee0e54e30ab25" exitCode=0
Jul 29 23:09:53.824147 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.823681    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"
Jul 29 23:09:53.824147 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.823694    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88" event={"ID":"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4","Type":"ContainerDied","Data":"ff91d2cac25745ac783eaede38047ae6cea06c7025c2c62c23aee0e54e30ab25"}
Jul 29 23:09:53.824147 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.823735    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88" event={"ID":"4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4","Type":"ContainerDied","Data":"f04144b0756dd98ecbe49a72fc5146d7a99598d5fa5c0afc646c60316e331d8b"}
Jul 29 23:09:53.824147 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.823756    2569 scope.go:117] "RemoveContainer" containerID="ff91d2cac25745ac783eaede38047ae6cea06c7025c2c62c23aee0e54e30ab25"
Jul 29 23:09:53.832401 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.832380    2569 scope.go:117] "RemoveContainer" containerID="ff91d2cac25745ac783eaede38047ae6cea06c7025c2c62c23aee0e54e30ab25"
Jul 29 23:09:53.832680 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:09:53.832657    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff91d2cac25745ac783eaede38047ae6cea06c7025c2c62c23aee0e54e30ab25\": container with ID starting with ff91d2cac25745ac783eaede38047ae6cea06c7025c2c62c23aee0e54e30ab25 not found: ID does not exist" containerID="ff91d2cac25745ac783eaede38047ae6cea06c7025c2c62c23aee0e54e30ab25"
Jul 29 23:09:53.832767 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.832686    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff91d2cac25745ac783eaede38047ae6cea06c7025c2c62c23aee0e54e30ab25"} err="failed to get container status \"ff91d2cac25745ac783eaede38047ae6cea06c7025c2c62c23aee0e54e30ab25\": rpc error: code = NotFound desc = could not find container \"ff91d2cac25745ac783eaede38047ae6cea06c7025c2c62c23aee0e54e30ab25\": container with ID starting with ff91d2cac25745ac783eaede38047ae6cea06c7025c2c62c23aee0e54e30ab25 not found: ID does not exist"
Jul 29 23:09:53.843778 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.843754    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"]
Jul 29 23:09:53.847580 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:53.847562    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g6n88"]
Jul 29 23:09:55.474683 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:09:55.474648    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4" path="/var/lib/kubelet/pods/4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4/volumes"
Jul 29 23:10:17.394986 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.394949    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"]
Jul 29 23:10:17.395455 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.395434    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4" containerName="main"
Jul 29 23:10:17.395510 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.395460    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4" containerName="main"
Jul 29 23:10:17.395609 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.395596    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cdb517f-bf5b-49a6-a4d1-4c7e4c1a3cb4" containerName="main"
Jul 29 23:10:17.398694 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.398668    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.400841 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.400817    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 29 23:10:17.401506 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.401472    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-sdgp8\""
Jul 29 23:10:17.401615 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.401548    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 29 23:10:17.401615 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.401575    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 29 23:10:17.406905 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.406882    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"]
Jul 29 23:10:17.472913 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.472878    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcvmk\" (UniqueName: \"kubernetes.io/projected/10c43d02-ff80-4c08-8bbb-ac679d14041e-kube-api-access-jcvmk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.473145 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.472922    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.473145 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.472953    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c43d02-ff80-4c08-8bbb-ac679d14041e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.473145 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.472984    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.473145 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.472999    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.473145 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.473074    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.574283 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.574245    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c43d02-ff80-4c08-8bbb-ac679d14041e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.574511 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.574302    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.574511 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.574408    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.574511 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.574457    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.574695 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.574525    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvmk\" (UniqueName: \"kubernetes.io/projected/10c43d02-ff80-4c08-8bbb-ac679d14041e-kube-api-access-jcvmk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.574695 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.574574    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.574695 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.574677    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.574843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.574762    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.574843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.574827    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.576832 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.576804    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.576933 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.576844    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c43d02-ff80-4c08-8bbb-ac679d14041e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.585909 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.585873    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvmk\" (UniqueName: \"kubernetes.io/projected/10c43d02-ff80-4c08-8bbb-ac679d14041e-kube-api-access-jcvmk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.710247 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.710164    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:17.838452 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.838422    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"]
Jul 29 23:10:17.840573 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:10:17.840546    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c43d02_ff80_4c08_8bbb_ac679d14041e.slice/crio-abb86b61ffb37627238088c39b089daeddadf4dfa08c685563a6cab27c054663 WatchSource:0}: Error finding container abb86b61ffb37627238088c39b089daeddadf4dfa08c685563a6cab27c054663: Status 404 returned error can't find the container with id abb86b61ffb37627238088c39b089daeddadf4dfa08c685563a6cab27c054663
Jul 29 23:10:17.915575 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.915537    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb" event={"ID":"10c43d02-ff80-4c08-8bbb-ac679d14041e","Type":"ContainerStarted","Data":"f92d0235f97407b30603bbcc43a0c7a50cd527e00b109b901140f39166e97a75"}
Jul 29 23:10:17.915742 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.915602    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb" event={"ID":"10c43d02-ff80-4c08-8bbb-ac679d14041e","Type":"ContainerStarted","Data":"abb86b61ffb37627238088c39b089daeddadf4dfa08c685563a6cab27c054663"}
Jul 29 23:10:17.931914 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:17.931845    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb" podStartSLOduration=0.931822812 podStartE2EDuration="931.822812ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:10:17.930665472 +0000 UTC m=+1424.935862896" watchObservedRunningTime="2026-07-29 23:10:17.931822812 +0000 UTC m=+1424.937020236"
Jul 29 23:10:18.711107 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:18.711062    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:27.710448 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:27.710362    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:27.722843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:27.722821    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:27.965838 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:27.965762    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:51.548894 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:51.548853    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"]
Jul 29 23:10:51.557793 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:10:51.557748    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:10:51.557793 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:10:51.557793 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:10:51.557793 ip-10-0-135-98 kubenswrapper[2569]:  > containerID="f92d0235f97407b30603bbcc43a0c7a50cd527e00b109b901140f39166e97a75" cmd=["/bin/sleep","15"]
Jul 29 23:10:51.558062 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:51.557818    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb" message="" err=<
Jul 29 23:10:51.558062 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:10:51.558062 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:10:51.558062 ip-10-0-135-98 kubenswrapper[2569]:  >
Jul 29 23:10:51.558062 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:10:51.557834    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:10:51.558062 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:10:51.558062 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:10:51.558062 ip-10-0-135-98 kubenswrapper[2569]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb" podUID="10c43d02-ff80-4c08-8bbb-ac679d14041e" containerName="main" containerID="cri-o://f92d0235f97407b30603bbcc43a0c7a50cd527e00b109b901140f39166e97a75"
Jul 29 23:10:51.558062 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:51.557880    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb" podUID="10c43d02-ff80-4c08-8bbb-ac679d14041e" containerName="main" containerID="cri-o://f92d0235f97407b30603bbcc43a0c7a50cd527e00b109b901140f39166e97a75" gracePeriod=60
Jul 29 23:10:51.955275 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:51.955178    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb" podUID="10c43d02-ff80-4c08-8bbb-ac679d14041e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 29 23:10:52.050850 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.050814    2569 generic.go:356] "Generic (PLEG): container finished" podID="10c43d02-ff80-4c08-8bbb-ac679d14041e" containerID="f92d0235f97407b30603bbcc43a0c7a50cd527e00b109b901140f39166e97a75" exitCode=0
Jul 29 23:10:52.051011 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.050868    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb" event={"ID":"10c43d02-ff80-4c08-8bbb-ac679d14041e","Type":"ContainerDied","Data":"f92d0235f97407b30603bbcc43a0c7a50cd527e00b109b901140f39166e97a75"}
Jul 29 23:10:52.290782 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.290758    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:52.368275 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.368244    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-dshm\") pod \"10c43d02-ff80-4c08-8bbb-ac679d14041e\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") "
Jul 29 23:10:52.368444 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.368285    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c43d02-ff80-4c08-8bbb-ac679d14041e-tls-certs\") pod \"10c43d02-ff80-4c08-8bbb-ac679d14041e\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") "
Jul 29 23:10:52.368444 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.368351    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-home\") pod \"10c43d02-ff80-4c08-8bbb-ac679d14041e\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") "
Jul 29 23:10:52.368444 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.368376    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcvmk\" (UniqueName: \"kubernetes.io/projected/10c43d02-ff80-4c08-8bbb-ac679d14041e-kube-api-access-jcvmk\") pod \"10c43d02-ff80-4c08-8bbb-ac679d14041e\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") "
Jul 29 23:10:52.368444 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.368405    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-model-cache\") pod \"10c43d02-ff80-4c08-8bbb-ac679d14041e\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") "
Jul 29 23:10:52.368682 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.368504    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-tmp-dir\") pod \"10c43d02-ff80-4c08-8bbb-ac679d14041e\" (UID: \"10c43d02-ff80-4c08-8bbb-ac679d14041e\") "
Jul 29 23:10:52.368682 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.368652    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-home" (OuterVolumeSpecName: "home") pod "10c43d02-ff80-4c08-8bbb-ac679d14041e" (UID: "10c43d02-ff80-4c08-8bbb-ac679d14041e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:10:52.368682 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.368663    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-model-cache" (OuterVolumeSpecName: "model-cache") pod "10c43d02-ff80-4c08-8bbb-ac679d14041e" (UID: "10c43d02-ff80-4c08-8bbb-ac679d14041e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:10:52.368834 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.368773    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:10:52.368834 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.368765    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10c43d02-ff80-4c08-8bbb-ac679d14041e" (UID: "10c43d02-ff80-4c08-8bbb-ac679d14041e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:10:52.368834 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.368786    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:10:52.370522 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.370496    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-dshm" (OuterVolumeSpecName: "dshm") pod "10c43d02-ff80-4c08-8bbb-ac679d14041e" (UID: "10c43d02-ff80-4c08-8bbb-ac679d14041e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:10:52.370630 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.370556    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c43d02-ff80-4c08-8bbb-ac679d14041e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10c43d02-ff80-4c08-8bbb-ac679d14041e" (UID: "10c43d02-ff80-4c08-8bbb-ac679d14041e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:10:52.370630 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.370575    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c43d02-ff80-4c08-8bbb-ac679d14041e-kube-api-access-jcvmk" (OuterVolumeSpecName: "kube-api-access-jcvmk") pod "10c43d02-ff80-4c08-8bbb-ac679d14041e" (UID: "10c43d02-ff80-4c08-8bbb-ac679d14041e"). InnerVolumeSpecName "kube-api-access-jcvmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:10:52.469633 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.469598    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:10:52.469633 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.469628    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c43d02-ff80-4c08-8bbb-ac679d14041e-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:10:52.469814 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.469643    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c43d02-ff80-4c08-8bbb-ac679d14041e-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:10:52.469814 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:52.469653    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcvmk\" (UniqueName: \"kubernetes.io/projected/10c43d02-ff80-4c08-8bbb-ac679d14041e-kube-api-access-jcvmk\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:10:53.056888 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:53.056850    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb" event={"ID":"10c43d02-ff80-4c08-8bbb-ac679d14041e","Type":"ContainerDied","Data":"abb86b61ffb37627238088c39b089daeddadf4dfa08c685563a6cab27c054663"}
Jul 29 23:10:53.056888 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:53.056885    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"
Jul 29 23:10:53.057416 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:53.056914    2569 scope.go:117] "RemoveContainer" containerID="f92d0235f97407b30603bbcc43a0c7a50cd527e00b109b901140f39166e97a75"
Jul 29 23:10:53.076946 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:53.076919    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"]
Jul 29 23:10:53.082233 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:53.082210    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fgprxb"]
Jul 29 23:10:53.474085 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:10:53.474055    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c43d02-ff80-4c08-8bbb-ac679d14041e" path="/var/lib/kubelet/pods/10c43d02-ff80-4c08-8bbb-ac679d14041e/volumes"
Jul 29 23:11:28.378938 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.378907    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"]
Jul 29 23:11:28.379435 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.379264    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c43d02-ff80-4c08-8bbb-ac679d14041e" containerName="main"
Jul 29 23:11:28.379435 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.379278    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c43d02-ff80-4c08-8bbb-ac679d14041e" containerName="main"
Jul 29 23:11:28.379435 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.379341    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c43d02-ff80-4c08-8bbb-ac679d14041e" containerName="main"
Jul 29 23:11:28.382025 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.382010    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.384394 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.384372    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 29 23:11:28.384538 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.384451    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 29 23:11:28.385052 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.385026    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 29 23:11:28.385121 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.385029    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-rwhd7\""
Jul 29 23:11:28.391138 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.391116    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"]
Jul 29 23:11:28.568818 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.568777    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.569008 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.568920    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.569377 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.568958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.569657 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.569633    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.569845 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.569829    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.569973 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.569958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6jdk\" (UniqueName: \"kubernetes.io/projected/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-kube-api-access-q6jdk\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.670731 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.670648    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jdk\" (UniqueName: \"kubernetes.io/projected/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-kube-api-access-q6jdk\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.670731 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.670701    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.670946 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.670756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.670946 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.670779    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.670946 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.670838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.670946 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.670875    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.671274 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.671241    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.671417 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.671274    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.671417 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.671322    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.673174 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.673149    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.673281 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.673262    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.682120 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.682094    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jdk\" (UniqueName: \"kubernetes.io/projected/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-kube-api-access-q6jdk\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.692756 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.692734    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:28.826135 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:28.826110    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"]
Jul 29 23:11:28.828204 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:11:28.828178    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eaaa294_a1ea_4d8d_99c4_23b37b98454e.slice/crio-626bb403f4940e14d462f195fd8101dc78632ad706618cbdb597493eef2a5949 WatchSource:0}: Error finding container 626bb403f4940e14d462f195fd8101dc78632ad706618cbdb597493eef2a5949: Status 404 returned error can't find the container with id 626bb403f4940e14d462f195fd8101dc78632ad706618cbdb597493eef2a5949
Jul 29 23:11:29.196962 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:29.196924    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9" event={"ID":"6eaaa294-a1ea-4d8d-99c4-23b37b98454e","Type":"ContainerStarted","Data":"82b2e6fe4be4ad8e97b5d836ac11a7834c8c9d066a0dfdd2ae2bb619cbfabf9e"}
Jul 29 23:11:29.197162 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:29.196976    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9" event={"ID":"6eaaa294-a1ea-4d8d-99c4-23b37b98454e","Type":"ContainerStarted","Data":"626bb403f4940e14d462f195fd8101dc78632ad706618cbdb597493eef2a5949"}
Jul 29 23:11:29.218655 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:29.218607    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9" podStartSLOduration=1.21859156 podStartE2EDuration="1.21859156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:11:29.21662988 +0000 UTC m=+1496.221827303" watchObservedRunningTime="2026-07-29 23:11:29.21859156 +0000 UTC m=+1496.223788984"
Jul 29 23:11:29.693802 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:29.693768    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:33.522844 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:33.522812    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:11:33.527281 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:33.527257    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:11:38.693436 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:38.693398    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:38.705659 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:38.705628    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:11:39.250977 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:11:39.250939    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:12:03.025624 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.025537    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"]
Jul 29 23:12:03.037046 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:12:03.037010    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:12:03.037046 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:12:03.037046 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:12:03.037046 ip-10-0-135-98 kubenswrapper[2569]:  > containerID="82b2e6fe4be4ad8e97b5d836ac11a7834c8c9d066a0dfdd2ae2bb619cbfabf9e" cmd=["/bin/sleep","15"]
Jul 29 23:12:03.037276 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.037064    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9" message="" err=<
Jul 29 23:12:03.037276 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:12:03.037276 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:12:03.037276 ip-10-0-135-98 kubenswrapper[2569]:  >
Jul 29 23:12:03.037276 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:12:03.037077    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:12:03.037276 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:12:03.037276 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:12:03.037276 ip-10-0-135-98 kubenswrapper[2569]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9" podUID="6eaaa294-a1ea-4d8d-99c4-23b37b98454e" containerName="main" containerID="cri-o://82b2e6fe4be4ad8e97b5d836ac11a7834c8c9d066a0dfdd2ae2bb619cbfabf9e"
Jul 29 23:12:03.037276 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.037127    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9" podUID="6eaaa294-a1ea-4d8d-99c4-23b37b98454e" containerName="main" containerID="cri-o://82b2e6fe4be4ad8e97b5d836ac11a7834c8c9d066a0dfdd2ae2bb619cbfabf9e" gracePeriod=60
Jul 29 23:12:03.240598 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.240548    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9" podUID="6eaaa294-a1ea-4d8d-99c4-23b37b98454e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 29 23:12:03.335550 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.335509    2569 generic.go:356] "Generic (PLEG): container finished" podID="6eaaa294-a1ea-4d8d-99c4-23b37b98454e" containerID="82b2e6fe4be4ad8e97b5d836ac11a7834c8c9d066a0dfdd2ae2bb619cbfabf9e" exitCode=0
Jul 29 23:12:03.335694 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.335516    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9" event={"ID":"6eaaa294-a1ea-4d8d-99c4-23b37b98454e","Type":"ContainerDied","Data":"82b2e6fe4be4ad8e97b5d836ac11a7834c8c9d066a0dfdd2ae2bb619cbfabf9e"}
Jul 29 23:12:03.384947 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.384922    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:12:03.459159 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.459129    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-model-cache\") pod \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") "
Jul 29 23:12:03.459350 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.459192    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-dshm\") pod \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") "
Jul 29 23:12:03.459350 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.459222    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-home\") pod \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") "
Jul 29 23:12:03.459350 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.459251    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-tmp-dir\") pod \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") "
Jul 29 23:12:03.459350 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.459287    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-tls-certs\") pod \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") "
Jul 29 23:12:03.459350 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.459337    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6jdk\" (UniqueName: \"kubernetes.io/projected/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-kube-api-access-q6jdk\") pod \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\" (UID: \"6eaaa294-a1ea-4d8d-99c4-23b37b98454e\") "
Jul 29 23:12:03.459647 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.459434    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-model-cache" (OuterVolumeSpecName: "model-cache") pod "6eaaa294-a1ea-4d8d-99c4-23b37b98454e" (UID: "6eaaa294-a1ea-4d8d-99c4-23b37b98454e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:12:03.459647 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.459498    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-home" (OuterVolumeSpecName: "home") pod "6eaaa294-a1ea-4d8d-99c4-23b37b98454e" (UID: "6eaaa294-a1ea-4d8d-99c4-23b37b98454e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:12:03.459647 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.459587    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6eaaa294-a1ea-4d8d-99c4-23b37b98454e" (UID: "6eaaa294-a1ea-4d8d-99c4-23b37b98454e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:12:03.459647 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.459614    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:12:03.461314 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.461290    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-dshm" (OuterVolumeSpecName: "dshm") pod "6eaaa294-a1ea-4d8d-99c4-23b37b98454e" (UID: "6eaaa294-a1ea-4d8d-99c4-23b37b98454e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:12:03.461386 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.461369    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6eaaa294-a1ea-4d8d-99c4-23b37b98454e" (UID: "6eaaa294-a1ea-4d8d-99c4-23b37b98454e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:12:03.461524 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.461497    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-kube-api-access-q6jdk" (OuterVolumeSpecName: "kube-api-access-q6jdk") pod "6eaaa294-a1ea-4d8d-99c4-23b37b98454e" (UID: "6eaaa294-a1ea-4d8d-99c4-23b37b98454e"). InnerVolumeSpecName "kube-api-access-q6jdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:12:03.560815 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.560727    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:12:03.560815 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.560759    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6jdk\" (UniqueName: \"kubernetes.io/projected/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-kube-api-access-q6jdk\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:12:03.560815 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.560774    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:12:03.560815 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.560785    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:12:03.561059 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:03.560822    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaaa294-a1ea-4d8d-99c4-23b37b98454e-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:12:04.340575 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:04.340530    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"
Jul 29 23:12:04.341077 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:04.340532    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9" event={"ID":"6eaaa294-a1ea-4d8d-99c4-23b37b98454e","Type":"ContainerDied","Data":"626bb403f4940e14d462f195fd8101dc78632ad706618cbdb597493eef2a5949"}
Jul 29 23:12:04.341077 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:04.340695    2569 scope.go:117] "RemoveContainer" containerID="82b2e6fe4be4ad8e97b5d836ac11a7834c8c9d066a0dfdd2ae2bb619cbfabf9e"
Jul 29 23:12:04.354980 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:04.354954    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"]
Jul 29 23:12:04.360168 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:04.360146    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-qvht9"]
Jul 29 23:12:05.473616 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:05.473582    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eaaa294-a1ea-4d8d-99c4-23b37b98454e" path="/var/lib/kubelet/pods/6eaaa294-a1ea-4d8d-99c4-23b37b98454e/volumes"
Jul 29 23:12:29.495785 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.495743    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"]
Jul 29 23:12:29.496265 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.496192    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eaaa294-a1ea-4d8d-99c4-23b37b98454e" containerName="main"
Jul 29 23:12:29.496265 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.496213    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eaaa294-a1ea-4d8d-99c4-23b37b98454e" containerName="main"
Jul 29 23:12:29.496380 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.496298    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eaaa294-a1ea-4d8d-99c4-23b37b98454e" containerName="main"
Jul 29 23:12:29.500604 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.500580    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.502638 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.502614    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 29 23:12:29.503228 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.503209    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-xbw9l\""
Jul 29 23:12:29.503290 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.503228    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 29 23:12:29.503290 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.503232    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 29 23:12:29.507042 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.507012    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"]
Jul 29 23:12:29.687088 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.687051    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.687280 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.687093    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d82z5\" (UniqueName: \"kubernetes.io/projected/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-kube-api-access-d82z5\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.687280 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.687156    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.687280 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.687233    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.687280 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.687270    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.687579 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.687337    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.688435 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.688410    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"]
Jul 29 23:12:29.691612 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.691593    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"
Jul 29 23:12:29.693749 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.693727    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-gqn8x\""
Jul 29 23:12:29.700628 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.700598    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"]
Jul 29 23:12:29.787964 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.787863    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.787964 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.787918    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.787964 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.787943    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d82z5\" (UniqueName: \"kubernetes.io/projected/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-kube-api-access-d82z5\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.788260 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.787994    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.788260 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.788040    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.788260 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.788064    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.788423 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.788281    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.788423 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.788323    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.788423 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.788372    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.790315 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.790294    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.790732 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.790711    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.796682 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.796656    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d82z5\" (UniqueName: \"kubernetes.io/projected/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-kube-api-access-d82z5\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-c679l\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.812393 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.812363    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:29.888994 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.888903    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjfmz\" (UniqueName: \"kubernetes.io/projected/21682b44-a114-4d98-8324-da534fb10e90-kube-api-access-zjfmz\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh\" (UID: \"21682b44-a114-4d98-8324-da534fb10e90\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"
Jul 29 23:12:29.888994 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.888949    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21682b44-a114-4d98-8324-da534fb10e90-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh\" (UID: \"21682b44-a114-4d98-8324-da534fb10e90\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"
Jul 29 23:12:29.948687 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.948594    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"]
Jul 29 23:12:29.951422 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:12:29.951393    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb0478a_32b1_4ee2_9b9a_03a4497d1c85.slice/crio-f9b730ff586035accabbee59340b9e728f6d0127682c746da2842645733be9ba WatchSource:0}: Error finding container f9b730ff586035accabbee59340b9e728f6d0127682c746da2842645733be9ba: Status 404 returned error can't find the container with id f9b730ff586035accabbee59340b9e728f6d0127682c746da2842645733be9ba
Jul 29 23:12:29.989997 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.989963    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjfmz\" (UniqueName: \"kubernetes.io/projected/21682b44-a114-4d98-8324-da534fb10e90-kube-api-access-zjfmz\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh\" (UID: \"21682b44-a114-4d98-8324-da534fb10e90\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"
Jul 29 23:12:29.990172 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.990008    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21682b44-a114-4d98-8324-da534fb10e90-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh\" (UID: \"21682b44-a114-4d98-8324-da534fb10e90\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"
Jul 29 23:12:29.992675 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.992647    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21682b44-a114-4d98-8324-da534fb10e90-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh\" (UID: \"21682b44-a114-4d98-8324-da534fb10e90\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"
Jul 29 23:12:29.997042 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:29.997020    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjfmz\" (UniqueName: \"kubernetes.io/projected/21682b44-a114-4d98-8324-da534fb10e90-kube-api-access-zjfmz\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh\" (UID: \"21682b44-a114-4d98-8324-da534fb10e90\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"
Jul 29 23:12:30.003901 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:30.003878    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"
Jul 29 23:12:30.142056 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:30.142017    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"]
Jul 29 23:12:30.144803 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:12:30.144775    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21682b44_a114_4d98_8324_da534fb10e90.slice/crio-d51a692163accd72cc62f15fe3077ad1f088605629c1b62446c8255e2f5c62d8 WatchSource:0}: Error finding container d51a692163accd72cc62f15fe3077ad1f088605629c1b62446c8255e2f5c62d8: Status 404 returned error can't find the container with id d51a692163accd72cc62f15fe3077ad1f088605629c1b62446c8255e2f5c62d8
Jul 29 23:12:30.146873 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:30.146856    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 23:12:30.442794 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:30.442709    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh" event={"ID":"21682b44-a114-4d98-8324-da534fb10e90","Type":"ContainerStarted","Data":"d51a692163accd72cc62f15fe3077ad1f088605629c1b62446c8255e2f5c62d8"}
Jul 29 23:12:30.444115 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:30.444092    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l" event={"ID":"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85","Type":"ContainerStarted","Data":"af9808c6b382e81744dcd47a969ef0112698db092ce323353688b45e3c835861"}
Jul 29 23:12:30.444215 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:30.444130    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l" event={"ID":"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85","Type":"ContainerStarted","Data":"f9b730ff586035accabbee59340b9e728f6d0127682c746da2842645733be9ba"}
Jul 29 23:12:30.462784 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:30.462730    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l" podStartSLOduration=1.462712312 podStartE2EDuration="1.462712312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:12:30.46101806 +0000 UTC m=+1557.466215484" watchObservedRunningTime="2026-07-29 23:12:30.462712312 +0000 UTC m=+1557.467909735"
Jul 29 23:12:30.812987 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:30.812945    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:33.460965 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:33.460930    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh" event={"ID":"21682b44-a114-4d98-8324-da534fb10e90","Type":"ContainerStarted","Data":"b067e00598a8ef767ab6c13235f23362f8d75fc17ea01e72d73d35c45532ff71"}
Jul 29 23:12:33.461450 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:33.461050    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"
Jul 29 23:12:33.481917 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:33.481855    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh" podStartSLOduration=2.059388261 podStartE2EDuration="4.48183375s" totalImagesPullingTime="2.422445489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:12:33.478056364 +0000 UTC m=+1560.483253777" watchObservedRunningTime="2026-07-29 23:12:33.48183375 +0000 UTC m=+1560.487031175"
Jul 29 23:12:39.813205 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:39.813162    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:39.825746 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:39.825713    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:12:40.498032 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:12:40.498002    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:13:04.466934 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:04.466900    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"
Jul 29 23:13:05.677447 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:05.677401    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"]
Jul 29 23:13:05.688992 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:05.688958    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"]
Jul 29 23:13:05.698254 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:13:05.698210    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:13:05.698254 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:13:05.698254 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:13:05.698254 ip-10-0-135-98 kubenswrapper[2569]:  > containerID="af9808c6b382e81744dcd47a969ef0112698db092ce323353688b45e3c835861" cmd=["/bin/sleep","15"]
Jul 29 23:13:05.698622 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:05.698273    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l" message="" err=<
Jul 29 23:13:05.698622 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:13:05.698622 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:13:05.698622 ip-10-0-135-98 kubenswrapper[2569]:  >
Jul 29 23:13:05.698622 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:13:05.698292    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:13:05.698622 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:13:05.698622 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:13:05.698622 ip-10-0-135-98 kubenswrapper[2569]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l" podUID="ccb0478a-32b1-4ee2-9b9a-03a4497d1c85" containerName="main" containerID="cri-o://af9808c6b382e81744dcd47a969ef0112698db092ce323353688b45e3c835861"
Jul 29 23:13:05.698622 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:05.698333    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l" podUID="ccb0478a-32b1-4ee2-9b9a-03a4497d1c85" containerName="main" containerID="cri-o://af9808c6b382e81744dcd47a969ef0112698db092ce323353688b45e3c835861" gracePeriod=60
Jul 29 23:13:06.050758 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.050724    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:13:06.193812 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.193770    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-home\") pod \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") "
Jul 29 23:13:06.193812 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.193816    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-model-cache\") pod \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") "
Jul 29 23:13:06.194083 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.193859    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d82z5\" (UniqueName: \"kubernetes.io/projected/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-kube-api-access-d82z5\") pod \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") "
Jul 29 23:13:06.194083 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.193928    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-dshm\") pod \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") "
Jul 29 23:13:06.194083 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.193950    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-tls-certs\") pod \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") "
Jul 29 23:13:06.194083 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.194000    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-tmp-dir\") pod \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\" (UID: \"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85\") "
Jul 29 23:13:06.194256 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.194172    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-model-cache" (OuterVolumeSpecName: "model-cache") pod "ccb0478a-32b1-4ee2-9b9a-03a4497d1c85" (UID: "ccb0478a-32b1-4ee2-9b9a-03a4497d1c85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:13:06.194753 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.194145    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-home" (OuterVolumeSpecName: "home") pod "ccb0478a-32b1-4ee2-9b9a-03a4497d1c85" (UID: "ccb0478a-32b1-4ee2-9b9a-03a4497d1c85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:13:06.194905 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.194839    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ccb0478a-32b1-4ee2-9b9a-03a4497d1c85" (UID: "ccb0478a-32b1-4ee2-9b9a-03a4497d1c85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:13:06.195186 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.194961    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:13:06.197188 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.197062    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-dshm" (OuterVolumeSpecName: "dshm") pod "ccb0478a-32b1-4ee2-9b9a-03a4497d1c85" (UID: "ccb0478a-32b1-4ee2-9b9a-03a4497d1c85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:13:06.199540 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.197821    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-kube-api-access-d82z5" (OuterVolumeSpecName: "kube-api-access-d82z5") pod "ccb0478a-32b1-4ee2-9b9a-03a4497d1c85" (UID: "ccb0478a-32b1-4ee2-9b9a-03a4497d1c85"). InnerVolumeSpecName "kube-api-access-d82z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:13:06.202340 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.202307    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ccb0478a-32b1-4ee2-9b9a-03a4497d1c85" (UID: "ccb0478a-32b1-4ee2-9b9a-03a4497d1c85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:13:06.296350 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.296306    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:13:06.296350 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.296341    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:13:06.296350 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.296352    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:13:06.296350 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.296363    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:13:06.296652 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.296373    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d82z5\" (UniqueName: \"kubernetes.io/projected/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85-kube-api-access-d82z5\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:13:06.593278 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.593189    2569 generic.go:356] "Generic (PLEG): container finished" podID="ccb0478a-32b1-4ee2-9b9a-03a4497d1c85" containerID="af9808c6b382e81744dcd47a969ef0112698db092ce323353688b45e3c835861" exitCode=0
Jul 29 23:13:06.593278 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.593232    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l" event={"ID":"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85","Type":"ContainerDied","Data":"af9808c6b382e81744dcd47a969ef0112698db092ce323353688b45e3c835861"}
Jul 29 23:13:06.593278 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.593261    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"
Jul 29 23:13:06.593278 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.593276    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l" event={"ID":"ccb0478a-32b1-4ee2-9b9a-03a4497d1c85","Type":"ContainerDied","Data":"f9b730ff586035accabbee59340b9e728f6d0127682c746da2842645733be9ba"}
Jul 29 23:13:06.593629 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.593298    2569 scope.go:117] "RemoveContainer" containerID="af9808c6b382e81744dcd47a969ef0112698db092ce323353688b45e3c835861"
Jul 29 23:13:06.602252 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.602122    2569 scope.go:117] "RemoveContainer" containerID="af9808c6b382e81744dcd47a969ef0112698db092ce323353688b45e3c835861"
Jul 29 23:13:06.602440 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:13:06.602419    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9808c6b382e81744dcd47a969ef0112698db092ce323353688b45e3c835861\": container with ID starting with af9808c6b382e81744dcd47a969ef0112698db092ce323353688b45e3c835861 not found: ID does not exist" containerID="af9808c6b382e81744dcd47a969ef0112698db092ce323353688b45e3c835861"
Jul 29 23:13:06.602518 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.602449    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9808c6b382e81744dcd47a969ef0112698db092ce323353688b45e3c835861"} err="failed to get container status \"af9808c6b382e81744dcd47a969ef0112698db092ce323353688b45e3c835861\": rpc error: code = NotFound desc = could not find container \"af9808c6b382e81744dcd47a969ef0112698db092ce323353688b45e3c835861\": container with ID starting with af9808c6b382e81744dcd47a969ef0112698db092ce323353688b45e3c835861 not found: ID does not exist"
Jul 29 23:13:06.614474 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.614443    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"]
Jul 29 23:13:06.618799 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:06.618774    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-c679l"]
Jul 29 23:13:07.474338 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:07.474301    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccb0478a-32b1-4ee2-9b9a-03a4497d1c85" path="/var/lib/kubelet/pods/ccb0478a-32b1-4ee2-9b9a-03a4497d1c85/volumes"
Jul 29 23:13:14.466825 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:14.466786    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh" podUID="21682b44-a114-4d98-8324-da534fb10e90" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:13:16.853492 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:16.853404    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh" message="" err="command '/bin/sleep 15' exited with 137: "
Jul 29 23:13:16.853492 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:13:16.853442    2569 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh" podUID="21682b44-a114-4d98-8324-da534fb10e90" containerName="main" containerID="cri-o://b067e00598a8ef767ab6c13235f23362f8d75fc17ea01e72d73d35c45532ff71"
Jul 29 23:13:16.854038 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:16.853526    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh" podUID="21682b44-a114-4d98-8324-da534fb10e90" containerName="main" containerID="cri-o://b067e00598a8ef767ab6c13235f23362f8d75fc17ea01e72d73d35c45532ff71" gracePeriod=49
Jul 29 23:13:17.487513 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.487473    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh_21682b44-a114-4d98-8324-da534fb10e90/main/0.log"
Jul 29 23:13:17.489876 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.489859    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"
Jul 29 23:13:17.592396 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.592358    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21682b44-a114-4d98-8324-da534fb10e90-tls-certs\") pod \"21682b44-a114-4d98-8324-da534fb10e90\" (UID: \"21682b44-a114-4d98-8324-da534fb10e90\") "
Jul 29 23:13:17.592396 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.592403    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjfmz\" (UniqueName: \"kubernetes.io/projected/21682b44-a114-4d98-8324-da534fb10e90-kube-api-access-zjfmz\") pod \"21682b44-a114-4d98-8324-da534fb10e90\" (UID: \"21682b44-a114-4d98-8324-da534fb10e90\") "
Jul 29 23:13:17.594501 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.594459    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21682b44-a114-4d98-8324-da534fb10e90-kube-api-access-zjfmz" (OuterVolumeSpecName: "kube-api-access-zjfmz") pod "21682b44-a114-4d98-8324-da534fb10e90" (UID: "21682b44-a114-4d98-8324-da534fb10e90"). InnerVolumeSpecName "kube-api-access-zjfmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:13:17.594614 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.594473    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21682b44-a114-4d98-8324-da534fb10e90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21682b44-a114-4d98-8324-da534fb10e90" (UID: "21682b44-a114-4d98-8324-da534fb10e90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:13:17.640122 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.640046    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh_21682b44-a114-4d98-8324-da534fb10e90/main/0.log"
Jul 29 23:13:17.640122 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.640089    2569 generic.go:356] "Generic (PLEG): container finished" podID="21682b44-a114-4d98-8324-da534fb10e90" containerID="b067e00598a8ef767ab6c13235f23362f8d75fc17ea01e72d73d35c45532ff71" exitCode=1
Jul 29 23:13:17.640293 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.640134    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh" event={"ID":"21682b44-a114-4d98-8324-da534fb10e90","Type":"ContainerDied","Data":"b067e00598a8ef767ab6c13235f23362f8d75fc17ea01e72d73d35c45532ff71"}
Jul 29 23:13:17.640293 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.640162    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"
Jul 29 23:13:17.640293 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.640205    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh" event={"ID":"21682b44-a114-4d98-8324-da534fb10e90","Type":"ContainerDied","Data":"d51a692163accd72cc62f15fe3077ad1f088605629c1b62446c8255e2f5c62d8"}
Jul 29 23:13:17.640293 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.640236    2569 scope.go:117] "RemoveContainer" containerID="b067e00598a8ef767ab6c13235f23362f8d75fc17ea01e72d73d35c45532ff71"
Jul 29 23:13:17.648649 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.648628    2569 scope.go:117] "RemoveContainer" containerID="b067e00598a8ef767ab6c13235f23362f8d75fc17ea01e72d73d35c45532ff71"
Jul 29 23:13:17.648910 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:13:17.648888    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b067e00598a8ef767ab6c13235f23362f8d75fc17ea01e72d73d35c45532ff71\": container with ID starting with b067e00598a8ef767ab6c13235f23362f8d75fc17ea01e72d73d35c45532ff71 not found: ID does not exist" containerID="b067e00598a8ef767ab6c13235f23362f8d75fc17ea01e72d73d35c45532ff71"
Jul 29 23:13:17.648974 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.648924    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b067e00598a8ef767ab6c13235f23362f8d75fc17ea01e72d73d35c45532ff71"} err="failed to get container status \"b067e00598a8ef767ab6c13235f23362f8d75fc17ea01e72d73d35c45532ff71\": rpc error: code = NotFound desc = could not find container \"b067e00598a8ef767ab6c13235f23362f8d75fc17ea01e72d73d35c45532ff71\": container with ID starting with b067e00598a8ef767ab6c13235f23362f8d75fc17ea01e72d73d35c45532ff71 not found: ID does not exist"
Jul 29 23:13:17.659635 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.659607    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"]
Jul 29 23:13:17.663308 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.663287    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c486785l2vh"]
Jul 29 23:13:17.693359 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.693335    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21682b44-a114-4d98-8324-da534fb10e90-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:13:17.693359 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:17.693359    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjfmz\" (UniqueName: \"kubernetes.io/projected/21682b44-a114-4d98-8324-da534fb10e90-kube-api-access-zjfmz\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:13:19.474057 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:19.474025    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21682b44-a114-4d98-8324-da534fb10e90" path="/var/lib/kubelet/pods/21682b44-a114-4d98-8324-da534fb10e90/volumes"
Jul 29 23:13:33.197323 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.197242    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"]
Jul 29 23:13:33.197697 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.197570    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb0478a-32b1-4ee2-9b9a-03a4497d1c85" containerName="main"
Jul 29 23:13:33.197697 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.197586    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb0478a-32b1-4ee2-9b9a-03a4497d1c85" containerName="main"
Jul 29 23:13:33.197697 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.197598    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21682b44-a114-4d98-8324-da534fb10e90" containerName="main"
Jul 29 23:13:33.197697 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.197603    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="21682b44-a114-4d98-8324-da534fb10e90" containerName="main"
Jul 29 23:13:33.197697 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.197668    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb0478a-32b1-4ee2-9b9a-03a4497d1c85" containerName="main"
Jul 29 23:13:33.197697 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.197676    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="21682b44-a114-4d98-8324-da534fb10e90" containerName="main"
Jul 29 23:13:33.202051 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.202033    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.204733 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.204707    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 29 23:13:33.204874 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.204707    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-ntlvx\""
Jul 29 23:13:33.204874 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.204761    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 29 23:13:33.204874 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.204761    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 29 23:13:33.211158 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.211135    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"]
Jul 29 23:13:33.221205 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.221177    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.221320 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.221218    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5z2\" (UniqueName: \"kubernetes.io/projected/f76c6905-56dd-45df-be7d-840b17b35fdd-kube-api-access-vz5z2\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.221320 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.221242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.221320 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.221301    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.221432 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.221344    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f76c6905-56dd-45df-be7d-840b17b35fdd-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.221432 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.221365    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.322588 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.322550    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.322788 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.322603    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5z2\" (UniqueName: \"kubernetes.io/projected/f76c6905-56dd-45df-be7d-840b17b35fdd-kube-api-access-vz5z2\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.322788 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.322637    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.322788 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.322677    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.322788 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.322729    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f76c6905-56dd-45df-be7d-840b17b35fdd-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.322788 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.322759    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.323096 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.323048    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.323096 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.323090    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.323180 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.323167    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.324898 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.324876    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 29 23:13:33.325108 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.325091    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.330709 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.330688    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 29 23:13:33.335211 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.335189    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f76c6905-56dd-45df-be7d-840b17b35fdd-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.341057 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.341036    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 29 23:13:33.351889 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.351867    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5z2\" (UniqueName: \"kubernetes.io/projected/f76c6905-56dd-45df-be7d-840b17b35fdd-kube-api-access-vz5z2\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-ck4wg\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.435438 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.435406    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"]
Jul 29 23:13:33.438888 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.438865    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:13:33.440996 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.440968    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-8f8z6\""
Jul 29 23:13:33.449979 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.449929    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"]
Jul 29 23:13:33.519048 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.519018    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-ntlvx\""
Jul 29 23:13:33.523219 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.523205    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:33.524606 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.524589    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a928cfd9-8156-4db2-90b1-359328735d60-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:13:33.524677 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.524642    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8qgk\" (UniqueName: \"kubernetes.io/projected/a928cfd9-8156-4db2-90b1-359328735d60-kube-api-access-d8qgk\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:13:33.524677 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.524667    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/a928cfd9-8156-4db2-90b1-359328735d60-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:13:33.524786 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.524730    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a928cfd9-8156-4db2-90b1-359328735d60-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:13:33.626282 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.626248    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a928cfd9-8156-4db2-90b1-359328735d60-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:13:33.626423 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.626374    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8qgk\" (UniqueName: \"kubernetes.io/projected/a928cfd9-8156-4db2-90b1-359328735d60-kube-api-access-d8qgk\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:13:33.626474 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.626417    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/a928cfd9-8156-4db2-90b1-359328735d60-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:13:33.626474 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.626458    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a928cfd9-8156-4db2-90b1-359328735d60-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:13:33.626901 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.626866    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/a928cfd9-8156-4db2-90b1-359328735d60-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:13:33.626989 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.626877    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a928cfd9-8156-4db2-90b1-359328735d60-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:13:33.628868 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.628838    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a928cfd9-8156-4db2-90b1-359328735d60-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:13:33.634923 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.634893    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8qgk\" (UniqueName: \"kubernetes.io/projected/a928cfd9-8156-4db2-90b1-359328735d60-kube-api-access-d8qgk\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:13:33.656289 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.656268    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"]
Jul 29 23:13:33.658562 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:13:33.658537    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76c6905_56dd_45df_be7d_840b17b35fdd.slice/crio-4d7bfd8227773fb4233ce51698133ffb0d3fc5c7d0e99185349f6df2ac6f8382 WatchSource:0}: Error finding container 4d7bfd8227773fb4233ce51698133ffb0d3fc5c7d0e99185349f6df2ac6f8382: Status 404 returned error can't find the container with id 4d7bfd8227773fb4233ce51698133ffb0d3fc5c7d0e99185349f6df2ac6f8382
Jul 29 23:13:33.703638 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.703614    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg" event={"ID":"f76c6905-56dd-45df-be7d-840b17b35fdd","Type":"ContainerStarted","Data":"4d7bfd8227773fb4233ce51698133ffb0d3fc5c7d0e99185349f6df2ac6f8382"}
Jul 29 23:13:33.748964 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.748932    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:13:33.876658 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:33.876634    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"]
Jul 29 23:13:33.878711 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:13:33.878684    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda928cfd9_8156_4db2_90b1_359328735d60.slice/crio-963c66003ed21705253836fb09465f451e00f861f890df3d89c6ffc3e2d829ea WatchSource:0}: Error finding container 963c66003ed21705253836fb09465f451e00f861f890df3d89c6ffc3e2d829ea: Status 404 returned error can't find the container with id 963c66003ed21705253836fb09465f451e00f861f890df3d89c6ffc3e2d829ea
Jul 29 23:13:34.709085 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:34.709047    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p" event={"ID":"a928cfd9-8156-4db2-90b1-359328735d60","Type":"ContainerStarted","Data":"abef4fd81bc9dc92ea0084672732ca4980219c170de4974da2a3e8c4fc7cf81c"}
Jul 29 23:13:34.709570 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:34.709104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p" event={"ID":"a928cfd9-8156-4db2-90b1-359328735d60","Type":"ContainerStarted","Data":"963c66003ed21705253836fb09465f451e00f861f890df3d89c6ffc3e2d829ea"}
Jul 29 23:13:34.709570 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:34.709154    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:13:34.715944 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:34.715913    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg" event={"ID":"f76c6905-56dd-45df-be7d-840b17b35fdd","Type":"ContainerStarted","Data":"56d44e261bbcc5ee1a74a9c5605ce88e5717d3f99d0984cc882f5ab4c852feeb"}
Jul 29 23:13:34.731441 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:34.731374    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p" podStartSLOduration=1.731350349 podStartE2EDuration="1.731350349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:13:34.729101964 +0000 UTC m=+1621.734299388" watchObservedRunningTime="2026-07-29 23:13:34.731350349 +0000 UTC m=+1621.736547772"
Jul 29 23:13:34.748918 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:34.748858    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg" podStartSLOduration=1.748842138 podStartE2EDuration="1.748842138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:13:34.746816158 +0000 UTC m=+1621.752013580" watchObservedRunningTime="2026-07-29 23:13:34.748842138 +0000 UTC m=+1621.754039561"
Jul 29 23:13:35.524408 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:35.524376    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:43.523898 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:43.523859    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:43.536351 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:43.536322    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:13:43.764650 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:13:43.764621    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:14:05.722195 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:05.722167    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:14:06.873394 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:06.873360    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"]
Jul 29 23:14:06.875035 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:06.875001    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"]
Jul 29 23:14:06.884907 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:14:06.884870    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:14:06.884907 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:14:06.884907 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:14:06.884907 ip-10-0-135-98 kubenswrapper[2569]:  > containerID="56d44e261bbcc5ee1a74a9c5605ce88e5717d3f99d0984cc882f5ab4c852feeb" cmd=["/bin/sleep","15"]
Jul 29 23:14:06.885221 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:06.884922    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg" message="" err=<
Jul 29 23:14:06.885221 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:14:06.885221 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:14:06.885221 ip-10-0-135-98 kubenswrapper[2569]:  >
Jul 29 23:14:06.885221 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:14:06.884938    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:14:06.885221 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:14:06.885221 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:14:06.885221 ip-10-0-135-98 kubenswrapper[2569]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg" podUID="f76c6905-56dd-45df-be7d-840b17b35fdd" containerName="main" containerID="cri-o://56d44e261bbcc5ee1a74a9c5605ce88e5717d3f99d0984cc882f5ab4c852feeb"
Jul 29 23:14:06.885221 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:06.884971    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg" podUID="f76c6905-56dd-45df-be7d-840b17b35fdd" containerName="main" containerID="cri-o://56d44e261bbcc5ee1a74a9c5605ce88e5717d3f99d0984cc882f5ab4c852feeb" gracePeriod=60
Jul 29 23:14:07.224788 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.224760    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:14:07.327639 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.327606    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-tmp-dir\") pod \"f76c6905-56dd-45df-be7d-840b17b35fdd\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") "
Jul 29 23:14:07.327639 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.327638    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-model-cache\") pod \"f76c6905-56dd-45df-be7d-840b17b35fdd\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") "
Jul 29 23:14:07.327899 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.327661    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz5z2\" (UniqueName: \"kubernetes.io/projected/f76c6905-56dd-45df-be7d-840b17b35fdd-kube-api-access-vz5z2\") pod \"f76c6905-56dd-45df-be7d-840b17b35fdd\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") "
Jul 29 23:14:07.327899 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.327681    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f76c6905-56dd-45df-be7d-840b17b35fdd-tls-certs\") pod \"f76c6905-56dd-45df-be7d-840b17b35fdd\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") "
Jul 29 23:14:07.327899 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.327711    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-dshm\") pod \"f76c6905-56dd-45df-be7d-840b17b35fdd\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") "
Jul 29 23:14:07.327899 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.327769    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-home\") pod \"f76c6905-56dd-45df-be7d-840b17b35fdd\" (UID: \"f76c6905-56dd-45df-be7d-840b17b35fdd\") "
Jul 29 23:14:07.328183 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.327942    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-model-cache" (OuterVolumeSpecName: "model-cache") pod "f76c6905-56dd-45df-be7d-840b17b35fdd" (UID: "f76c6905-56dd-45df-be7d-840b17b35fdd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:14:07.328183 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.327954    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f76c6905-56dd-45df-be7d-840b17b35fdd" (UID: "f76c6905-56dd-45df-be7d-840b17b35fdd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:14:07.328183 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.328006    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-home" (OuterVolumeSpecName: "home") pod "f76c6905-56dd-45df-be7d-840b17b35fdd" (UID: "f76c6905-56dd-45df-be7d-840b17b35fdd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:14:07.328183 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.328086    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:14:07.328183 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.328103    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:14:07.328183 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.328113    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:14:07.329988 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.329955    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76c6905-56dd-45df-be7d-840b17b35fdd-kube-api-access-vz5z2" (OuterVolumeSpecName: "kube-api-access-vz5z2") pod "f76c6905-56dd-45df-be7d-840b17b35fdd" (UID: "f76c6905-56dd-45df-be7d-840b17b35fdd"). InnerVolumeSpecName "kube-api-access-vz5z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:14:07.330118 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.330000    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-dshm" (OuterVolumeSpecName: "dshm") pod "f76c6905-56dd-45df-be7d-840b17b35fdd" (UID: "f76c6905-56dd-45df-be7d-840b17b35fdd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:14:07.330118 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.330065    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76c6905-56dd-45df-be7d-840b17b35fdd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f76c6905-56dd-45df-be7d-840b17b35fdd" (UID: "f76c6905-56dd-45df-be7d-840b17b35fdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:14:07.429191 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.429106    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz5z2\" (UniqueName: \"kubernetes.io/projected/f76c6905-56dd-45df-be7d-840b17b35fdd-kube-api-access-vz5z2\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:14:07.429191 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.429145    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f76c6905-56dd-45df-be7d-840b17b35fdd-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:14:07.429191 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.429162    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f76c6905-56dd-45df-be7d-840b17b35fdd-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:14:07.851972 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.851929    2569 generic.go:356] "Generic (PLEG): container finished" podID="f76c6905-56dd-45df-be7d-840b17b35fdd" containerID="56d44e261bbcc5ee1a74a9c5605ce88e5717d3f99d0984cc882f5ab4c852feeb" exitCode=0
Jul 29 23:14:07.852151 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.851994    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"
Jul 29 23:14:07.852151 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.852013    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg" event={"ID":"f76c6905-56dd-45df-be7d-840b17b35fdd","Type":"ContainerDied","Data":"56d44e261bbcc5ee1a74a9c5605ce88e5717d3f99d0984cc882f5ab4c852feeb"}
Jul 29 23:14:07.852151 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.852076    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg" event={"ID":"f76c6905-56dd-45df-be7d-840b17b35fdd","Type":"ContainerDied","Data":"4d7bfd8227773fb4233ce51698133ffb0d3fc5c7d0e99185349f6df2ac6f8382"}
Jul 29 23:14:07.852151 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.852099    2569 scope.go:117] "RemoveContainer" containerID="56d44e261bbcc5ee1a74a9c5605ce88e5717d3f99d0984cc882f5ab4c852feeb"
Jul 29 23:14:07.860157 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.860136    2569 scope.go:117] "RemoveContainer" containerID="56d44e261bbcc5ee1a74a9c5605ce88e5717d3f99d0984cc882f5ab4c852feeb"
Jul 29 23:14:07.860434 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:14:07.860398    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56d44e261bbcc5ee1a74a9c5605ce88e5717d3f99d0984cc882f5ab4c852feeb\": container with ID starting with 56d44e261bbcc5ee1a74a9c5605ce88e5717d3f99d0984cc882f5ab4c852feeb not found: ID does not exist" containerID="56d44e261bbcc5ee1a74a9c5605ce88e5717d3f99d0984cc882f5ab4c852feeb"
Jul 29 23:14:07.860541 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.860441    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d44e261bbcc5ee1a74a9c5605ce88e5717d3f99d0984cc882f5ab4c852feeb"} err="failed to get container status \"56d44e261bbcc5ee1a74a9c5605ce88e5717d3f99d0984cc882f5ab4c852feeb\": rpc error: code = NotFound desc = could not find container \"56d44e261bbcc5ee1a74a9c5605ce88e5717d3f99d0984cc882f5ab4c852feeb\": container with ID starting with 56d44e261bbcc5ee1a74a9c5605ce88e5717d3f99d0984cc882f5ab4c852feeb not found: ID does not exist"
Jul 29 23:14:07.865771 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.865748    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"]
Jul 29 23:14:07.869248 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:07.869224    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-ck4wg"]
Jul 29 23:14:09.473545 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:09.473509    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f76c6905-56dd-45df-be7d-840b17b35fdd" path="/var/lib/kubelet/pods/f76c6905-56dd-45df-be7d-840b17b35fdd/volumes"
Jul 29 23:14:15.721708 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:15.721670    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p" podUID="a928cfd9-8156-4db2-90b1-359328735d60" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:14:21.894187 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:21.894139    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p" podUID="a928cfd9-8156-4db2-90b1-359328735d60" containerName="main" containerID="cri-o://abef4fd81bc9dc92ea0084672732ca4980219c170de4974da2a3e8c4fc7cf81c" gracePeriod=45
Jul 29 23:14:22.136698 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.136674    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:14:22.149352 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.149293    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a928cfd9-8156-4db2-90b1-359328735d60-tokenizer-cache\") pod \"a928cfd9-8156-4db2-90b1-359328735d60\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") "
Jul 29 23:14:22.149352 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.149328    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8qgk\" (UniqueName: \"kubernetes.io/projected/a928cfd9-8156-4db2-90b1-359328735d60-kube-api-access-d8qgk\") pod \"a928cfd9-8156-4db2-90b1-359328735d60\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") "
Jul 29 23:14:22.149528 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.149433    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/a928cfd9-8156-4db2-90b1-359328735d60-cachi2-cache\") pod \"a928cfd9-8156-4db2-90b1-359328735d60\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") "
Jul 29 23:14:22.149528 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.149474    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a928cfd9-8156-4db2-90b1-359328735d60-tls-certs\") pod \"a928cfd9-8156-4db2-90b1-359328735d60\" (UID: \"a928cfd9-8156-4db2-90b1-359328735d60\") "
Jul 29 23:14:22.149640 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.149556    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a928cfd9-8156-4db2-90b1-359328735d60-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "a928cfd9-8156-4db2-90b1-359328735d60" (UID: "a928cfd9-8156-4db2-90b1-359328735d60"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:14:22.149698 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.149651    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a928cfd9-8156-4db2-90b1-359328735d60-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "a928cfd9-8156-4db2-90b1-359328735d60" (UID: "a928cfd9-8156-4db2-90b1-359328735d60"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:14:22.149756 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.149738    2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a928cfd9-8156-4db2-90b1-359328735d60-tokenizer-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:14:22.149812 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.149758    2569 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/a928cfd9-8156-4db2-90b1-359328735d60-cachi2-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:14:22.151475 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.151440    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a928cfd9-8156-4db2-90b1-359328735d60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a928cfd9-8156-4db2-90b1-359328735d60" (UID: "a928cfd9-8156-4db2-90b1-359328735d60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:14:22.151475 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.151462    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a928cfd9-8156-4db2-90b1-359328735d60-kube-api-access-d8qgk" (OuterVolumeSpecName: "kube-api-access-d8qgk") pod "a928cfd9-8156-4db2-90b1-359328735d60" (UID: "a928cfd9-8156-4db2-90b1-359328735d60"). InnerVolumeSpecName "kube-api-access-d8qgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:14:22.250669 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.250632    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a928cfd9-8156-4db2-90b1-359328735d60-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:14:22.250669 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.250664    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8qgk\" (UniqueName: \"kubernetes.io/projected/a928cfd9-8156-4db2-90b1-359328735d60-kube-api-access-d8qgk\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:14:22.912830 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.912794    2569 generic.go:356] "Generic (PLEG): container finished" podID="a928cfd9-8156-4db2-90b1-359328735d60" containerID="abef4fd81bc9dc92ea0084672732ca4980219c170de4974da2a3e8c4fc7cf81c" exitCode=0
Jul 29 23:14:22.913346 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.912840    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"
Jul 29 23:14:22.913346 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.912883    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p" event={"ID":"a928cfd9-8156-4db2-90b1-359328735d60","Type":"ContainerDied","Data":"abef4fd81bc9dc92ea0084672732ca4980219c170de4974da2a3e8c4fc7cf81c"}
Jul 29 23:14:22.913346 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.912946    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p" event={"ID":"a928cfd9-8156-4db2-90b1-359328735d60","Type":"ContainerDied","Data":"963c66003ed21705253836fb09465f451e00f861f890df3d89c6ffc3e2d829ea"}
Jul 29 23:14:22.913346 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.912983    2569 scope.go:117] "RemoveContainer" containerID="abef4fd81bc9dc92ea0084672732ca4980219c170de4974da2a3e8c4fc7cf81c"
Jul 29 23:14:22.921882 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.921861    2569 scope.go:117] "RemoveContainer" containerID="abef4fd81bc9dc92ea0084672732ca4980219c170de4974da2a3e8c4fc7cf81c"
Jul 29 23:14:22.922147 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:14:22.922128    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abef4fd81bc9dc92ea0084672732ca4980219c170de4974da2a3e8c4fc7cf81c\": container with ID starting with abef4fd81bc9dc92ea0084672732ca4980219c170de4974da2a3e8c4fc7cf81c not found: ID does not exist" containerID="abef4fd81bc9dc92ea0084672732ca4980219c170de4974da2a3e8c4fc7cf81c"
Jul 29 23:14:22.922199 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.922158    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abef4fd81bc9dc92ea0084672732ca4980219c170de4974da2a3e8c4fc7cf81c"} err="failed to get container status \"abef4fd81bc9dc92ea0084672732ca4980219c170de4974da2a3e8c4fc7cf81c\": rpc error: code = NotFound desc = could not find container \"abef4fd81bc9dc92ea0084672732ca4980219c170de4974da2a3e8c4fc7cf81c\": container with ID starting with abef4fd81bc9dc92ea0084672732ca4980219c170de4974da2a3e8c4fc7cf81c not found: ID does not exist"
Jul 29 23:14:22.935315 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.935283    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"]
Jul 29 23:14:22.939120 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:22.939095    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7nr26p"]
Jul 29 23:14:23.474116 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:23.474080    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a928cfd9-8156-4db2-90b1-359328735d60" path="/var/lib/kubelet/pods/a928cfd9-8156-4db2-90b1-359328735d60/volumes"
Jul 29 23:14:29.700760 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.700720    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"]
Jul 29 23:14:29.701210 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.701175    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a928cfd9-8156-4db2-90b1-359328735d60" containerName="main"
Jul 29 23:14:29.701210 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.701194    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a928cfd9-8156-4db2-90b1-359328735d60" containerName="main"
Jul 29 23:14:29.701285 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.701214    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f76c6905-56dd-45df-be7d-840b17b35fdd" containerName="main"
Jul 29 23:14:29.701285 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.701223    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f76c6905-56dd-45df-be7d-840b17b35fdd" containerName="main"
Jul 29 23:14:29.701348 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.701316    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f76c6905-56dd-45df-be7d-840b17b35fdd" containerName="main"
Jul 29 23:14:29.701348 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.701330    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a928cfd9-8156-4db2-90b1-359328735d60" containerName="main"
Jul 29 23:14:29.706041 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.706020    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.708652 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.708625    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"openshift-service-ca.crt\""
Jul 29 23:14:29.708804 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.708712    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"kube-root-ca.crt\""
Jul 29 23:14:29.708804 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.708776    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-kserve-self-signed-certs\""
Jul 29 23:14:29.708933 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.708780    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"default-dockercfg-t27js\""
Jul 29 23:14:29.712739 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.712716    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"]
Jul 29 23:14:29.811031 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.810991    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.811214 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.811035    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.811214 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.811071    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s94vv\" (UniqueName: \"kubernetes.io/projected/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-kube-api-access-s94vv\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.811214 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.811113    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.811214 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.811138    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.811214 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.811157    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.911928 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.911892    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.912103 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.911947    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.912103 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.911976    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.912103 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.912013    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s94vv\" (UniqueName: \"kubernetes.io/projected/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-kube-api-access-s94vv\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.912226 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.912148    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.912226 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.912212    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.912325 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.912304    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.912366 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.912352    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.912526 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.912473    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.914405 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.914376    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.914589 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.914572    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:29.923916 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:29.923893    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s94vv\" (UniqueName: \"kubernetes.io/projected/ffadb9d9-e5f0-439a-b4e9-90bbe927b552-kube-api-access-s94vv\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf\" (UID: \"ffadb9d9-e5f0-439a-b4e9-90bbe927b552\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:30.017803 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:30.017761    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:30.154433 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:30.154407    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"]
Jul 29 23:14:30.157176 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:14:30.157145    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffadb9d9_e5f0_439a_b4e9_90bbe927b552.slice/crio-b689d46ab9d3163bc89c9ae6d4c1b6d615e7ca27cf31bf50eb6edae2e2167f74 WatchSource:0}: Error finding container b689d46ab9d3163bc89c9ae6d4c1b6d615e7ca27cf31bf50eb6edae2e2167f74: Status 404 returned error can't find the container with id b689d46ab9d3163bc89c9ae6d4c1b6d615e7ca27cf31bf50eb6edae2e2167f74
Jul 29 23:14:30.949084 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:30.949049    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf" event={"ID":"ffadb9d9-e5f0-439a-b4e9-90bbe927b552","Type":"ContainerStarted","Data":"346e872aaaee0bd4e66b5497433d84a350f70366a7bd728bd4139e0a3e520ae1"}
Jul 29 23:14:30.949513 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:30.949101    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf" event={"ID":"ffadb9d9-e5f0-439a-b4e9-90bbe927b552","Type":"ContainerStarted","Data":"b689d46ab9d3163bc89c9ae6d4c1b6d615e7ca27cf31bf50eb6edae2e2167f74"}
Jul 29 23:14:30.965653 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:30.965570    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf" podStartSLOduration=1.96555565 podStartE2EDuration="1.96555565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:14:30.96424725 +0000 UTC m=+1677.969444675" watchObservedRunningTime="2026-07-29 23:14:30.96555565 +0000 UTC m=+1677.970753073"
Jul 29 23:14:31.018861 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:31.018825    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:40.018173 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:40.018138    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:40.030823 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:40.030790    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:40.999114 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:40.999085    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-kg7kf"
Jul 29 23:14:57.047507 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.047406    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"]
Jul 29 23:14:57.052319 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.052295    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"
Jul 29 23:14:57.054820 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.054794    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-vx2lh\""
Jul 29 23:14:57.054959 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.054832    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 29 23:14:57.054959 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.054832    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-wvckp\""
Jul 29 23:14:57.054959 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.054897    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 29 23:14:57.054959 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.054906    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 29 23:14:57.060793 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.060766    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"]
Jul 29 23:14:57.151636 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.151596    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fac8673-fab9-419d-a8bb-decfc4e0ac09-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj\" (UID: \"7fac8673-fab9-419d-a8bb-decfc4e0ac09\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"
Jul 29 23:14:57.151823 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.151659    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9dhj\" (UniqueName: \"kubernetes.io/projected/7fac8673-fab9-419d-a8bb-decfc4e0ac09-kube-api-access-d9dhj\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj\" (UID: \"7fac8673-fab9-419d-a8bb-decfc4e0ac09\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"
Jul 29 23:14:57.252784 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.252747    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fac8673-fab9-419d-a8bb-decfc4e0ac09-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj\" (UID: \"7fac8673-fab9-419d-a8bb-decfc4e0ac09\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"
Jul 29 23:14:57.252953 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.252806    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9dhj\" (UniqueName: \"kubernetes.io/projected/7fac8673-fab9-419d-a8bb-decfc4e0ac09-kube-api-access-d9dhj\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj\" (UID: \"7fac8673-fab9-419d-a8bb-decfc4e0ac09\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"
Jul 29 23:14:57.255249 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.255224    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fac8673-fab9-419d-a8bb-decfc4e0ac09-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj\" (UID: \"7fac8673-fab9-419d-a8bb-decfc4e0ac09\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"
Jul 29 23:14:57.261097 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.261068    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9dhj\" (UniqueName: \"kubernetes.io/projected/7fac8673-fab9-419d-a8bb-decfc4e0ac09-kube-api-access-d9dhj\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj\" (UID: \"7fac8673-fab9-419d-a8bb-decfc4e0ac09\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"
Jul 29 23:14:57.362934 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.362845    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"
Jul 29 23:14:57.493017 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:57.492982    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"]
Jul 29 23:14:57.496141 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:14:57.496112    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fac8673_fab9_419d_a8bb_decfc4e0ac09.slice/crio-e584ea921c0bfe548c4c27f655b990f69ad00ee4e899f3ffd522820580d7d97e WatchSource:0}: Error finding container e584ea921c0bfe548c4c27f655b990f69ad00ee4e899f3ffd522820580d7d97e: Status 404 returned error can't find the container with id e584ea921c0bfe548c4c27f655b990f69ad00ee4e899f3ffd522820580d7d97e
Jul 29 23:14:58.056944 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:58.056912    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj" event={"ID":"7fac8673-fab9-419d-a8bb-decfc4e0ac09","Type":"ContainerStarted","Data":"e3a448fc63d374a37dbd220e4a22394d68308e053504f1321a32ff4eb024b23b"}
Jul 29 23:14:58.057344 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:58.056967    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj" event={"ID":"7fac8673-fab9-419d-a8bb-decfc4e0ac09","Type":"ContainerStarted","Data":"e584ea921c0bfe548c4c27f655b990f69ad00ee4e899f3ffd522820580d7d97e"}
Jul 29 23:14:58.057344 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:58.057049    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"
Jul 29 23:14:58.076943 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:14:58.076879    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj" podStartSLOduration=1.076859384 podStartE2EDuration="1.076859384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:14:58.076224311 +0000 UTC m=+1705.081421735" watchObservedRunningTime="2026-07-29 23:14:58.076859384 +0000 UTC m=+1705.082056808"
Jul 29 23:15:29.063558 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:15:29.063520    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"
Jul 29 23:16:30.457466 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:30.457387    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"]
Jul 29 23:16:33.549266 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:33.549236    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:16:33.555529 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:33.555503    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:16:39.062722 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:39.062677    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj" podUID="7fac8673-fab9-419d-a8bb-decfc4e0ac09" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:16:45.482073 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:45.482014    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj" podUID="7fac8673-fab9-419d-a8bb-decfc4e0ac09" containerName="main" containerID="cri-o://e3a448fc63d374a37dbd220e4a22394d68308e053504f1321a32ff4eb024b23b" gracePeriod=45
Jul 29 23:16:45.728797 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:45.728771    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"
Jul 29 23:16:45.843298 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:45.843204    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9dhj\" (UniqueName: \"kubernetes.io/projected/7fac8673-fab9-419d-a8bb-decfc4e0ac09-kube-api-access-d9dhj\") pod \"7fac8673-fab9-419d-a8bb-decfc4e0ac09\" (UID: \"7fac8673-fab9-419d-a8bb-decfc4e0ac09\") "
Jul 29 23:16:45.843298 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:45.843253    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fac8673-fab9-419d-a8bb-decfc4e0ac09-tls-certs\") pod \"7fac8673-fab9-419d-a8bb-decfc4e0ac09\" (UID: \"7fac8673-fab9-419d-a8bb-decfc4e0ac09\") "
Jul 29 23:16:45.845508 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:45.845461    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fac8673-fab9-419d-a8bb-decfc4e0ac09-kube-api-access-d9dhj" (OuterVolumeSpecName: "kube-api-access-d9dhj") pod "7fac8673-fab9-419d-a8bb-decfc4e0ac09" (UID: "7fac8673-fab9-419d-a8bb-decfc4e0ac09"). InnerVolumeSpecName "kube-api-access-d9dhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:16:45.845508 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:45.845499    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fac8673-fab9-419d-a8bb-decfc4e0ac09-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fac8673-fab9-419d-a8bb-decfc4e0ac09" (UID: "7fac8673-fab9-419d-a8bb-decfc4e0ac09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:16:45.944696 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:45.944658    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fac8673-fab9-419d-a8bb-decfc4e0ac09-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:16:45.944696 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:45.944690    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9dhj\" (UniqueName: \"kubernetes.io/projected/7fac8673-fab9-419d-a8bb-decfc4e0ac09-kube-api-access-d9dhj\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:16:46.492204 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:46.492163    2569 generic.go:356] "Generic (PLEG): container finished" podID="7fac8673-fab9-419d-a8bb-decfc4e0ac09" containerID="e3a448fc63d374a37dbd220e4a22394d68308e053504f1321a32ff4eb024b23b" exitCode=0
Jul 29 23:16:46.492204 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:46.492203    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"
Jul 29 23:16:46.492763 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:46.492204    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj" event={"ID":"7fac8673-fab9-419d-a8bb-decfc4e0ac09","Type":"ContainerDied","Data":"e3a448fc63d374a37dbd220e4a22394d68308e053504f1321a32ff4eb024b23b"}
Jul 29 23:16:46.492763 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:46.492326    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj" event={"ID":"7fac8673-fab9-419d-a8bb-decfc4e0ac09","Type":"ContainerDied","Data":"e584ea921c0bfe548c4c27f655b990f69ad00ee4e899f3ffd522820580d7d97e"}
Jul 29 23:16:46.492763 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:46.492354    2569 scope.go:117] "RemoveContainer" containerID="e3a448fc63d374a37dbd220e4a22394d68308e053504f1321a32ff4eb024b23b"
Jul 29 23:16:46.501525 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:46.501504    2569 scope.go:117] "RemoveContainer" containerID="e3a448fc63d374a37dbd220e4a22394d68308e053504f1321a32ff4eb024b23b"
Jul 29 23:16:46.501821 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:16:46.501802    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a448fc63d374a37dbd220e4a22394d68308e053504f1321a32ff4eb024b23b\": container with ID starting with e3a448fc63d374a37dbd220e4a22394d68308e053504f1321a32ff4eb024b23b not found: ID does not exist" containerID="e3a448fc63d374a37dbd220e4a22394d68308e053504f1321a32ff4eb024b23b"
Jul 29 23:16:46.501889 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:46.501829    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a448fc63d374a37dbd220e4a22394d68308e053504f1321a32ff4eb024b23b"} err="failed to get container status \"e3a448fc63d374a37dbd220e4a22394d68308e053504f1321a32ff4eb024b23b\": rpc error: code = NotFound desc = could not find container \"e3a448fc63d374a37dbd220e4a22394d68308e053504f1321a32ff4eb024b23b\": container with ID starting with e3a448fc63d374a37dbd220e4a22394d68308e053504f1321a32ff4eb024b23b not found: ID does not exist"
Jul 29 23:16:46.515640 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:46.515606    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"]
Jul 29 23:16:46.520811 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:46.520785    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-65m24zj"]
Jul 29 23:16:47.473931 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:47.473897    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fac8673-fab9-419d-a8bb-decfc4e0ac09" path="/var/lib/kubelet/pods/7fac8673-fab9-419d-a8bb-decfc4e0ac09/volumes"
Jul 29 23:16:55.370882 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.370843    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"]
Jul 29 23:16:55.371379 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.371292    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fac8673-fab9-419d-a8bb-decfc4e0ac09" containerName="main"
Jul 29 23:16:55.371379 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.371311    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fac8673-fab9-419d-a8bb-decfc4e0ac09" containerName="main"
Jul 29 23:16:55.371510 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.371413    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fac8673-fab9-419d-a8bb-decfc4e0ac09" containerName="main"
Jul 29 23:16:55.375667 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.375644    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"
Jul 29 23:16:55.378804 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.378777    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-sln8b\""
Jul 29 23:16:55.379012 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.378848    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 29 23:16:55.379012 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.378922    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 29 23:16:55.379139 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.379117    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-bhfrc\""
Jul 29 23:16:55.379189 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.379168    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 29 23:16:55.384409 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.384388    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"]
Jul 29 23:16:55.524495 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.524455    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7491c9d9-5f95-4542-a996-120d2d9519f3-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j\" (UID: \"7491c9d9-5f95-4542-a996-120d2d9519f3\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"
Jul 29 23:16:55.524666 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.524520    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8llgj\" (UniqueName: \"kubernetes.io/projected/7491c9d9-5f95-4542-a996-120d2d9519f3-kube-api-access-8llgj\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j\" (UID: \"7491c9d9-5f95-4542-a996-120d2d9519f3\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"
Jul 29 23:16:55.625846 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.625756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7491c9d9-5f95-4542-a996-120d2d9519f3-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j\" (UID: \"7491c9d9-5f95-4542-a996-120d2d9519f3\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"
Jul 29 23:16:55.625846 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.625805    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8llgj\" (UniqueName: \"kubernetes.io/projected/7491c9d9-5f95-4542-a996-120d2d9519f3-kube-api-access-8llgj\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j\" (UID: \"7491c9d9-5f95-4542-a996-120d2d9519f3\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"
Jul 29 23:16:55.628380 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.628350    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7491c9d9-5f95-4542-a996-120d2d9519f3-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j\" (UID: \"7491c9d9-5f95-4542-a996-120d2d9519f3\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"
Jul 29 23:16:55.633760 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.633726    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8llgj\" (UniqueName: \"kubernetes.io/projected/7491c9d9-5f95-4542-a996-120d2d9519f3-kube-api-access-8llgj\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j\" (UID: \"7491c9d9-5f95-4542-a996-120d2d9519f3\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"
Jul 29 23:16:55.687740 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.687702    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"
Jul 29 23:16:55.842821 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:55.842782    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"]
Jul 29 23:16:55.845209 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:16:55.845182    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7491c9d9_5f95_4542_a996_120d2d9519f3.slice/crio-92cf19c54bc815dc8df63710e39d5281be4c796f753a8162e9b3af04a63e907a WatchSource:0}: Error finding container 92cf19c54bc815dc8df63710e39d5281be4c796f753a8162e9b3af04a63e907a: Status 404 returned error can't find the container with id 92cf19c54bc815dc8df63710e39d5281be4c796f753a8162e9b3af04a63e907a
Jul 29 23:16:56.534542 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:56.534498    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j" event={"ID":"7491c9d9-5f95-4542-a996-120d2d9519f3","Type":"ContainerStarted","Data":"6280618c5f5b66dd2b968102f2af27d31db547b4e5f5b8a9061088331122c0b1"}
Jul 29 23:16:56.534984 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:56.534562    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j" event={"ID":"7491c9d9-5f95-4542-a996-120d2d9519f3","Type":"ContainerStarted","Data":"92cf19c54bc815dc8df63710e39d5281be4c796f753a8162e9b3af04a63e907a"}
Jul 29 23:16:56.534984 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:56.534682    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"
Jul 29 23:16:56.554113 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:16:56.554057    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j" podStartSLOduration=1.554039783 podStartE2EDuration="1.554039783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:16:56.552294376 +0000 UTC m=+1823.557491791" watchObservedRunningTime="2026-07-29 23:16:56.554039783 +0000 UTC m=+1823.559237207"
Jul 29 23:17:27.541576 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:27.541537    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"
Jul 29 23:17:28.896545 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:28.896512    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"]
Jul 29 23:17:37.541173 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:37.541134    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j" podUID="7491c9d9-5f95-4542-a996-120d2d9519f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:17:43.921027 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:43.920983    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j" podUID="7491c9d9-5f95-4542-a996-120d2d9519f3" containerName="main" containerID="cri-o://6280618c5f5b66dd2b968102f2af27d31db547b4e5f5b8a9061088331122c0b1" gracePeriod=45
Jul 29 23:17:44.168741 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.168709    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"
Jul 29 23:17:44.248751 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.248710    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8llgj\" (UniqueName: \"kubernetes.io/projected/7491c9d9-5f95-4542-a996-120d2d9519f3-kube-api-access-8llgj\") pod \"7491c9d9-5f95-4542-a996-120d2d9519f3\" (UID: \"7491c9d9-5f95-4542-a996-120d2d9519f3\") "
Jul 29 23:17:44.248960 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.248784    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7491c9d9-5f95-4542-a996-120d2d9519f3-tls-certs\") pod \"7491c9d9-5f95-4542-a996-120d2d9519f3\" (UID: \"7491c9d9-5f95-4542-a996-120d2d9519f3\") "
Jul 29 23:17:44.250928 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.250889    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7491c9d9-5f95-4542-a996-120d2d9519f3-kube-api-access-8llgj" (OuterVolumeSpecName: "kube-api-access-8llgj") pod "7491c9d9-5f95-4542-a996-120d2d9519f3" (UID: "7491c9d9-5f95-4542-a996-120d2d9519f3"). InnerVolumeSpecName "kube-api-access-8llgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:17:44.250928 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.250903    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7491c9d9-5f95-4542-a996-120d2d9519f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7491c9d9-5f95-4542-a996-120d2d9519f3" (UID: "7491c9d9-5f95-4542-a996-120d2d9519f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:17:44.349556 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.349513    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8llgj\" (UniqueName: \"kubernetes.io/projected/7491c9d9-5f95-4542-a996-120d2d9519f3-kube-api-access-8llgj\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:17:44.349556 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.349548    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7491c9d9-5f95-4542-a996-120d2d9519f3-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:17:44.728524 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.728459    2569 generic.go:356] "Generic (PLEG): container finished" podID="7491c9d9-5f95-4542-a996-120d2d9519f3" containerID="6280618c5f5b66dd2b968102f2af27d31db547b4e5f5b8a9061088331122c0b1" exitCode=0
Jul 29 23:17:44.728524 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.728521    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"
Jul 29 23:17:44.728747 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.728549    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j" event={"ID":"7491c9d9-5f95-4542-a996-120d2d9519f3","Type":"ContainerDied","Data":"6280618c5f5b66dd2b968102f2af27d31db547b4e5f5b8a9061088331122c0b1"}
Jul 29 23:17:44.728747 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.728613    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j" event={"ID":"7491c9d9-5f95-4542-a996-120d2d9519f3","Type":"ContainerDied","Data":"92cf19c54bc815dc8df63710e39d5281be4c796f753a8162e9b3af04a63e907a"}
Jul 29 23:17:44.728747 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.728642    2569 scope.go:117] "RemoveContainer" containerID="6280618c5f5b66dd2b968102f2af27d31db547b4e5f5b8a9061088331122c0b1"
Jul 29 23:17:44.737887 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.737867    2569 scope.go:117] "RemoveContainer" containerID="6280618c5f5b66dd2b968102f2af27d31db547b4e5f5b8a9061088331122c0b1"
Jul 29 23:17:44.738155 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:17:44.738131    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6280618c5f5b66dd2b968102f2af27d31db547b4e5f5b8a9061088331122c0b1\": container with ID starting with 6280618c5f5b66dd2b968102f2af27d31db547b4e5f5b8a9061088331122c0b1 not found: ID does not exist" containerID="6280618c5f5b66dd2b968102f2af27d31db547b4e5f5b8a9061088331122c0b1"
Jul 29 23:17:44.738197 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.738168    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6280618c5f5b66dd2b968102f2af27d31db547b4e5f5b8a9061088331122c0b1"} err="failed to get container status \"6280618c5f5b66dd2b968102f2af27d31db547b4e5f5b8a9061088331122c0b1\": rpc error: code = NotFound desc = could not find container \"6280618c5f5b66dd2b968102f2af27d31db547b4e5f5b8a9061088331122c0b1\": container with ID starting with 6280618c5f5b66dd2b968102f2af27d31db547b4e5f5b8a9061088331122c0b1 not found: ID does not exist"
Jul 29 23:17:44.749032 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.749008    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"]
Jul 29 23:17:44.752450 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:44.752431    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedm6n9j"]
Jul 29 23:17:45.473831 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:17:45.473795    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7491c9d9-5f95-4542-a996-120d2d9519f3" path="/var/lib/kubelet/pods/7491c9d9-5f95-4542-a996-120d2d9519f3/volumes"
Jul 29 23:18:04.597282 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.597196    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"]
Jul 29 23:18:04.597803 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.597522    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7491c9d9-5f95-4542-a996-120d2d9519f3" containerName="main"
Jul 29 23:18:04.597803 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.597535    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7491c9d9-5f95-4542-a996-120d2d9519f3" containerName="main"
Jul 29 23:18:04.597803 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.597606    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7491c9d9-5f95-4542-a996-120d2d9519f3" containerName="main"
Jul 29 23:18:04.602117 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.602095    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.604178 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.604145    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 29 23:18:04.604306 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.604180    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 29 23:18:04.604306 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.604256    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 29 23:18:04.604306 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.604299    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 29 23:18:04.604683 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.604662    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-rtb5v\""
Jul 29 23:18:04.610769 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.610740    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"]
Jul 29 23:18:04.626888 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.626848    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-model-cache\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.626888 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.626884    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6187990c-b60e-4602-b176-46676bab619d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.627107 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.626910    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtvtc\" (UniqueName: \"kubernetes.io/projected/6187990c-b60e-4602-b176-46676bab619d-kube-api-access-rtvtc\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.627107 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.626984    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.627107 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.627012    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-home\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.627107 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.627031    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-dshm\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.627238 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.627117    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.727761 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.727722    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-model-cache\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.727761 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.727760    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6187990c-b60e-4602-b176-46676bab619d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.728035 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.727791    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtvtc\" (UniqueName: \"kubernetes.io/projected/6187990c-b60e-4602-b176-46676bab619d-kube-api-access-rtvtc\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.728035 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.727830    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.728035 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.727851    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-home\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.728035 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.727867    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-dshm\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.728035 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.727936    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.728296 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.728272    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-home\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.728367 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.728285    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-model-cache\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.728367 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.728331    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.728465 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.728420    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.730306 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.730283    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-dshm\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.730506 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.730469    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6187990c-b60e-4602-b176-46676bab619d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.734887 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.734863    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtvtc\" (UniqueName: \"kubernetes.io/projected/6187990c-b60e-4602-b176-46676bab619d-kube-api-access-rtvtc\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:04.915309 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:04.915223    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:05.049680 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:05.049651    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"]
Jul 29 23:18:05.052116 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:18:05.052086    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6187990c_b60e_4602_b176_46676bab619d.slice/crio-80c7e7f4252813012934eca85b247cc7aab824f8f2ac8849cef07b5517f179c1 WatchSource:0}: Error finding container 80c7e7f4252813012934eca85b247cc7aab824f8f2ac8849cef07b5517f179c1: Status 404 returned error can't find the container with id 80c7e7f4252813012934eca85b247cc7aab824f8f2ac8849cef07b5517f179c1
Jul 29 23:18:05.820146 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:05.820108    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd" event={"ID":"6187990c-b60e-4602-b176-46676bab619d","Type":"ContainerStarted","Data":"7250ac3acad65e0243ea2dc64f9d07bce7794c42a7272ce8ad0c74d6f8b2eda1"}
Jul 29 23:18:05.820661 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:05.820163    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd" event={"ID":"6187990c-b60e-4602-b176-46676bab619d","Type":"ContainerStarted","Data":"80c7e7f4252813012934eca85b247cc7aab824f8f2ac8849cef07b5517f179c1"}
Jul 29 23:18:10.846687 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:10.846652    2569 generic.go:356] "Generic (PLEG): container finished" podID="6187990c-b60e-4602-b176-46676bab619d" containerID="7250ac3acad65e0243ea2dc64f9d07bce7794c42a7272ce8ad0c74d6f8b2eda1" exitCode=0
Jul 29 23:18:10.847091 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:10.846701    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd" event={"ID":"6187990c-b60e-4602-b176-46676bab619d","Type":"ContainerDied","Data":"7250ac3acad65e0243ea2dc64f9d07bce7794c42a7272ce8ad0c74d6f8b2eda1"}
Jul 29 23:18:11.853344 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:11.853303    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd" event={"ID":"6187990c-b60e-4602-b176-46676bab619d","Type":"ContainerStarted","Data":"9381ee58acc92eb2d8c611725d46ac637f0a4b43cc407d962901d72f4e22c514"}
Jul 29 23:18:11.873122 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:11.873072    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd" podStartSLOduration=2.934551413 podStartE2EDuration="7.873056122s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.938504709s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:18:11.872262547 +0000 UTC m=+1898.877459972" watchObservedRunningTime="2026-07-29 23:18:11.873056122 +0000 UTC m=+1898.878253544"
Jul 29 23:18:11.915587 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:11.915551    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:14.916211 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:14.916170    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:14.928604 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:14.928579    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:15.880922 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:15.880888    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:18:27.004585 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.004547    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"]
Jul 29 23:18:27.006940 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.006917    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.016062 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.016027    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"]
Jul 29 23:18:27.127953 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.127917    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.128143 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.127981    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-ca\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.128143 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.128023    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-hub\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.128143 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.128064    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.128143 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.128091    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7tst\" (UniqueName: \"kubernetes.io/projected/b21a2e37-ab9b-4e89-847f-cfbf19564217-kube-api-access-b7tst\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.128143 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.128119    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b21a2e37-ab9b-4e89-847f-cfbf19564217-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.229028 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.228982    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.229028 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.229030    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7tst\" (UniqueName: \"kubernetes.io/projected/b21a2e37-ab9b-4e89-847f-cfbf19564217-kube-api-access-b7tst\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.229306 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.229067    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b21a2e37-ab9b-4e89-847f-cfbf19564217-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.229306 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.229120    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.229306 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.229210    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-ca\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.229306 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.229245    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-hub\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.230043 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.230010    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b21a2e37-ab9b-4e89-847f-cfbf19564217-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.231651 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.231620    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-ca\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.231926 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.231902    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.231926 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.231918    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.232005 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.231919    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-hub\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.237726 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.237697    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7tst\" (UniqueName: \"kubernetes.io/projected/b21a2e37-ab9b-4e89-847f-cfbf19564217-kube-api-access-b7tst\") pod \"cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.316603 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.316473    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:27.456062 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.456028    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"]
Jul 29 23:18:27.458195 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:18:27.458160    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb21a2e37_ab9b_4e89_847f_cfbf19564217.slice/crio-e71592274810754d29e5012448e917711265b8162743c5cd8872af5027305005 WatchSource:0}: Error finding container e71592274810754d29e5012448e917711265b8162743c5cd8872af5027305005: Status 404 returned error can't find the container with id e71592274810754d29e5012448e917711265b8162743c5cd8872af5027305005
Jul 29 23:18:27.460416 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.460395    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 23:18:27.919902 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:27.919868    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq" event={"ID":"b21a2e37-ab9b-4e89-847f-cfbf19564217","Type":"ContainerStarted","Data":"e71592274810754d29e5012448e917711265b8162743c5cd8872af5027305005"}
Jul 29 23:18:31.944304 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:31.944264    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq" event={"ID":"b21a2e37-ab9b-4e89-847f-cfbf19564217","Type":"ContainerStarted","Data":"7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f"}
Jul 29 23:18:31.944304 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:31.944311    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq" event={"ID":"b21a2e37-ab9b-4e89-847f-cfbf19564217","Type":"ContainerStarted","Data":"3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581"}
Jul 29 23:18:31.944827 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:31.944324    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq" event={"ID":"b21a2e37-ab9b-4e89-847f-cfbf19564217","Type":"ContainerStarted","Data":"bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007"}
Jul 29 23:18:31.972628 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:31.972576    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq" podStartSLOduration=2.491559087 podStartE2EDuration="5.972560421s" totalImagesPullingTime="3.481001334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:18:31.97122789 +0000 UTC m=+1918.976425313" watchObservedRunningTime="2026-07-29 23:18:31.972560421 +0000 UTC m=+1918.977757844"
Jul 29 23:18:32.014437 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.014400    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"]
Jul 29 23:18:32.014644 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.014608    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="proxy-agent" containerID="cri-o://7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55" gracePeriod=30
Jul 29 23:18:32.014718 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.014656    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="addon-agent" containerID="cri-o://dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04" gracePeriod=30
Jul 29 23:18:32.014815 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.014659    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="service-proxy" containerID="cri-o://62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7" gracePeriod=30
Jul 29 23:18:32.267475 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.267450    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 23:18:32.380392 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.380353    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/968c59fb-1819-4280-8671-cd892f0d060a-additional-service-ca\") pod \"968c59fb-1819-4280-8671-cd892f0d060a\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") "
Jul 29 23:18:32.380630 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.380406    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-service-proxy-server-cert\") pod \"968c59fb-1819-4280-8671-cd892f0d060a\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") "
Jul 29 23:18:32.380630 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.380459    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-ca\") pod \"968c59fb-1819-4280-8671-cd892f0d060a\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") "
Jul 29 23:18:32.380630 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.380515    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-hub\") pod \"968c59fb-1819-4280-8671-cd892f0d060a\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") "
Jul 29 23:18:32.380630 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.380557    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flpdd\" (UniqueName: \"kubernetes.io/projected/968c59fb-1819-4280-8671-cd892f0d060a-kube-api-access-flpdd\") pod \"968c59fb-1819-4280-8671-cd892f0d060a\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") "
Jul 29 23:18:32.380630 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.380588    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-hub-kubeconfig\") pod \"968c59fb-1819-4280-8671-cd892f0d060a\" (UID: \"968c59fb-1819-4280-8671-cd892f0d060a\") "
Jul 29 23:18:32.380831 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.380780    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/968c59fb-1819-4280-8671-cd892f0d060a-additional-service-ca" (OuterVolumeSpecName: "additional-service-ca") pod "968c59fb-1819-4280-8671-cd892f0d060a" (UID: "968c59fb-1819-4280-8671-cd892f0d060a"). InnerVolumeSpecName "additional-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 23:18:32.382954 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.382921    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968c59fb-1819-4280-8671-cd892f0d060a-kube-api-access-flpdd" (OuterVolumeSpecName: "kube-api-access-flpdd") pod "968c59fb-1819-4280-8671-cd892f0d060a" (UID: "968c59fb-1819-4280-8671-cd892f0d060a"). InnerVolumeSpecName "kube-api-access-flpdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:18:32.382954 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.382941    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-hub" (OuterVolumeSpecName: "hub") pod "968c59fb-1819-4280-8671-cd892f0d060a" (UID: "968c59fb-1819-4280-8671-cd892f0d060a"). InnerVolumeSpecName "hub". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:18:32.383149 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.382991    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-hub-kubeconfig" (OuterVolumeSpecName: "hub-kubeconfig") pod "968c59fb-1819-4280-8671-cd892f0d060a" (UID: "968c59fb-1819-4280-8671-cd892f0d060a"). InnerVolumeSpecName "hub-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:18:32.383149 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.383117    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-ca" (OuterVolumeSpecName: "ca") pod "968c59fb-1819-4280-8671-cd892f0d060a" (UID: "968c59fb-1819-4280-8671-cd892f0d060a"). InnerVolumeSpecName "ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:18:32.383251 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.383203    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-service-proxy-server-cert" (OuterVolumeSpecName: "service-proxy-server-cert") pod "968c59fb-1819-4280-8671-cd892f0d060a" (UID: "968c59fb-1819-4280-8671-cd892f0d060a"). InnerVolumeSpecName "service-proxy-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:18:32.481659 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.481622    2569 reconciler_common.go:299] "Volume detached for volume \"ca\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:18:32.481659 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.481652    2569 reconciler_common.go:299] "Volume detached for volume \"hub\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-hub\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:18:32.481659 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.481663    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flpdd\" (UniqueName: \"kubernetes.io/projected/968c59fb-1819-4280-8671-cd892f0d060a-kube-api-access-flpdd\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:18:32.481900 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.481675    2569 reconciler_common.go:299] "Volume detached for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-hub-kubeconfig\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:18:32.481900 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.481684    2569 reconciler_common.go:299] "Volume detached for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/968c59fb-1819-4280-8671-cd892f0d060a-additional-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:18:32.481900 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.481692    2569 reconciler_common.go:299] "Volume detached for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/968c59fb-1819-4280-8671-cd892f0d060a-service-proxy-server-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:18:32.950383 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.950349    2569 generic.go:356] "Generic (PLEG): container finished" podID="968c59fb-1819-4280-8671-cd892f0d060a" containerID="62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7" exitCode=2
Jul 29 23:18:32.950383 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.950377    2569 generic.go:356] "Generic (PLEG): container finished" podID="968c59fb-1819-4280-8671-cd892f0d060a" containerID="dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04" exitCode=2
Jul 29 23:18:32.950383 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.950384    2569 generic.go:356] "Generic (PLEG): container finished" podID="968c59fb-1819-4280-8671-cd892f0d060a" containerID="7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55" exitCode=2
Jul 29 23:18:32.951002 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.950406    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"
Jul 29 23:18:32.951002 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.950434    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" event={"ID":"968c59fb-1819-4280-8671-cd892f0d060a","Type":"ContainerDied","Data":"62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7"}
Jul 29 23:18:32.951002 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.950504    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" event={"ID":"968c59fb-1819-4280-8671-cd892f0d060a","Type":"ContainerDied","Data":"dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04"}
Jul 29 23:18:32.951002 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.950524    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" event={"ID":"968c59fb-1819-4280-8671-cd892f0d060a","Type":"ContainerDied","Data":"7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55"}
Jul 29 23:18:32.951002 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.950536    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447" event={"ID":"968c59fb-1819-4280-8671-cd892f0d060a","Type":"ContainerDied","Data":"cebac003a21e038609c9b2af0e5324354576bac7388e60bb7627c5b899902b3c"}
Jul 29 23:18:32.951002 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.950553    2569 scope.go:117] "RemoveContainer" containerID="62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7"
Jul 29 23:18:32.959007 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.958986    2569 scope.go:117] "RemoveContainer" containerID="73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd"
Jul 29 23:18:32.966445 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.966421    2569 scope.go:117] "RemoveContainer" containerID="dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04"
Jul 29 23:18:32.973911 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.973890    2569 scope.go:117] "RemoveContainer" containerID="7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55"
Jul 29 23:18:32.974091 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.974069    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"]
Jul 29 23:18:32.980751 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.980727    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dd8d8f59c-hl447"]
Jul 29 23:18:32.981582 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.981563    2569 scope.go:117] "RemoveContainer" containerID="62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7"
Jul 29 23:18:32.981825 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:18:32.981808    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7\": container with ID starting with 62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7 not found: ID does not exist" containerID="62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7"
Jul 29 23:18:32.981875 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.981844    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7"} err="failed to get container status \"62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7\": rpc error: code = NotFound desc = could not find container \"62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7\": container with ID starting with 62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7 not found: ID does not exist"
Jul 29 23:18:32.981875 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.981864    2569 scope.go:117] "RemoveContainer" containerID="73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd"
Jul 29 23:18:32.982102 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:18:32.982087    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd\": container with ID starting with 73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd not found: ID does not exist" containerID="73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd"
Jul 29 23:18:32.982154 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.982106    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd"} err="failed to get container status \"73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd\": rpc error: code = NotFound desc = could not find container \"73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd\": container with ID starting with 73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd not found: ID does not exist"
Jul 29 23:18:32.982154 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.982125    2569 scope.go:117] "RemoveContainer" containerID="dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04"
Jul 29 23:18:32.982334 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:18:32.982317    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04\": container with ID starting with dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04 not found: ID does not exist" containerID="dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04"
Jul 29 23:18:32.982377 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.982340    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04"} err="failed to get container status \"dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04\": rpc error: code = NotFound desc = could not find container \"dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04\": container with ID starting with dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04 not found: ID does not exist"
Jul 29 23:18:32.982377 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.982359    2569 scope.go:117] "RemoveContainer" containerID="7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55"
Jul 29 23:18:32.982610 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:18:32.982593    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55\": container with ID starting with 7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55 not found: ID does not exist" containerID="7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55"
Jul 29 23:18:32.982683 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.982614    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55"} err="failed to get container status \"7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55\": rpc error: code = NotFound desc = could not find container \"7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55\": container with ID starting with 7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55 not found: ID does not exist"
Jul 29 23:18:32.982683 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.982629    2569 scope.go:117] "RemoveContainer" containerID="62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7"
Jul 29 23:18:32.982843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.982826    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7"} err="failed to get container status \"62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7\": rpc error: code = NotFound desc = could not find container \"62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7\": container with ID starting with 62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7 not found: ID does not exist"
Jul 29 23:18:32.982885 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.982844    2569 scope.go:117] "RemoveContainer" containerID="73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd"
Jul 29 23:18:32.983091 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.983071    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd"} err="failed to get container status \"73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd\": rpc error: code = NotFound desc = could not find container \"73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd\": container with ID starting with 73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd not found: ID does not exist"
Jul 29 23:18:32.983132 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.983093    2569 scope.go:117] "RemoveContainer" containerID="dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04"
Jul 29 23:18:32.983320 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.983303    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04"} err="failed to get container status \"dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04\": rpc error: code = NotFound desc = could not find container \"dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04\": container with ID starting with dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04 not found: ID does not exist"
Jul 29 23:18:32.983362 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.983320    2569 scope.go:117] "RemoveContainer" containerID="7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55"
Jul 29 23:18:32.983537 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.983520    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55"} err="failed to get container status \"7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55\": rpc error: code = NotFound desc = could not find container \"7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55\": container with ID starting with 7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55 not found: ID does not exist"
Jul 29 23:18:32.983599 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.983538    2569 scope.go:117] "RemoveContainer" containerID="62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7"
Jul 29 23:18:32.983739 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.983723    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7"} err="failed to get container status \"62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7\": rpc error: code = NotFound desc = could not find container \"62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7\": container with ID starting with 62090cd375b10edba57ab21c922d3e7f63f860cb0e3add9bd1fcbea7f0a790d7 not found: ID does not exist"
Jul 29 23:18:32.983779 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.983740    2569 scope.go:117] "RemoveContainer" containerID="73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd"
Jul 29 23:18:32.983942 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.983926    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd"} err="failed to get container status \"73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd\": rpc error: code = NotFound desc = could not find container \"73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd\": container with ID starting with 73b28a550f24bf8691cc8c1e2bb412141c066adce2e0dbc30540b0c5fae5cccd not found: ID does not exist"
Jul 29 23:18:32.983982 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.983943    2569 scope.go:117] "RemoveContainer" containerID="dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04"
Jul 29 23:18:32.984179 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.984162    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04"} err="failed to get container status \"dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04\": rpc error: code = NotFound desc = could not find container \"dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04\": container with ID starting with dff42acabd26a2b2189eea5182222fdf1c71d7c769a59b0c7b5711203a016b04 not found: ID does not exist"
Jul 29 23:18:32.984222 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.984180    2569 scope.go:117] "RemoveContainer" containerID="7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55"
Jul 29 23:18:32.984373 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:32.984356    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55"} err="failed to get container status \"7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55\": rpc error: code = NotFound desc = could not find container \"7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55\": container with ID starting with 7dc37bc53b6242d680926b109b4b98e3bb47fd79ad67932c028cba24c3a8bf55 not found: ID does not exist"
Jul 29 23:18:33.474092 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:33.474055    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="968c59fb-1819-4280-8671-cd892f0d060a" path="/var/lib/kubelet/pods/968c59fb-1819-4280-8671-cd892f0d060a/volumes"
Jul 29 23:18:53.226183 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.226145    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"]
Jul 29 23:18:53.226606 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.226458    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="service-proxy"
Jul 29 23:18:53.226606 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.226469    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="service-proxy"
Jul 29 23:18:53.226606 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.226492    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="proxy-agent"
Jul 29 23:18:53.226606 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.226498    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="proxy-agent"
Jul 29 23:18:53.226606 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.226509    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="addon-agent"
Jul 29 23:18:53.226606 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.226514    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="addon-agent"
Jul 29 23:18:53.226606 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.226579    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="proxy-agent"
Jul 29 23:18:53.226606 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.226587    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="addon-agent"
Jul 29 23:18:53.226606 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.226595    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="service-proxy"
Jul 29 23:18:53.226606 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.226600    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="service-proxy"
Jul 29 23:18:53.226934 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.226652    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="service-proxy"
Jul 29 23:18:53.226934 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.226657    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="968c59fb-1819-4280-8671-cd892f0d060a" containerName="service-proxy"
Jul 29 23:18:53.228545 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.228528    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.237824 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.237800    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"]
Jul 29 23:18:53.258920 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.258883    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f76edb-55b1-4add-b386-9d8d942bf929-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.258920 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.258932    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/54f76edb-55b1-4add-b386-9d8d942bf929-ca\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.259142 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.258951    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54f76edb-55b1-4add-b386-9d8d942bf929-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.259142 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.258976    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/54f76edb-55b1-4add-b386-9d8d942bf929-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.259142 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.259044    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/54f76edb-55b1-4add-b386-9d8d942bf929-hub\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.259142 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.259109    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb5zz\" (UniqueName: \"kubernetes.io/projected/54f76edb-55b1-4add-b386-9d8d942bf929-kube-api-access-mb5zz\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.359538 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.359496    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb5zz\" (UniqueName: \"kubernetes.io/projected/54f76edb-55b1-4add-b386-9d8d942bf929-kube-api-access-mb5zz\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.359724 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.359586    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f76edb-55b1-4add-b386-9d8d942bf929-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.359724 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.359654    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/54f76edb-55b1-4add-b386-9d8d942bf929-ca\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.359724 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.359683    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54f76edb-55b1-4add-b386-9d8d942bf929-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.359724 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.359709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/54f76edb-55b1-4add-b386-9d8d942bf929-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.359950 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.359745    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/54f76edb-55b1-4add-b386-9d8d942bf929-hub\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.360413 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.360379    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f76edb-55b1-4add-b386-9d8d942bf929-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.362350 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.362315    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/54f76edb-55b1-4add-b386-9d8d942bf929-hub\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.362516 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.362350    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54f76edb-55b1-4add-b386-9d8d942bf929-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.362603 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.362557    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/54f76edb-55b1-4add-b386-9d8d942bf929-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.362658 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.362595    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/54f76edb-55b1-4add-b386-9d8d942bf929-ca\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.367339 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.367314    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb5zz\" (UniqueName: \"kubernetes.io/projected/54f76edb-55b1-4add-b386-9d8d942bf929-kube-api-access-mb5zz\") pod \"cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z\" (UID: \"54f76edb-55b1-4add-b386-9d8d942bf929\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.538167 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.538069    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"
Jul 29 23:18:53.676526 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:53.676466    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z"]
Jul 29 23:18:53.680281 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:18:53.680246    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54f76edb_55b1_4add_b386_9d8d942bf929.slice/crio-120dc31bdc7d91c046d5e9749f10b7a3a9d59db5799c09005488d46060e8d82e WatchSource:0}: Error finding container 120dc31bdc7d91c046d5e9749f10b7a3a9d59db5799c09005488d46060e8d82e: Status 404 returned error can't find the container with id 120dc31bdc7d91c046d5e9749f10b7a3a9d59db5799c09005488d46060e8d82e
Jul 29 23:18:54.032698 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.032657    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z" event={"ID":"54f76edb-55b1-4add-b386-9d8d942bf929","Type":"ContainerStarted","Data":"8f0d8081d356e0f7e2171faea63cd3bfc6179575c7de0147254132132a91bc1d"}
Jul 29 23:18:54.032698 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.032702    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z" event={"ID":"54f76edb-55b1-4add-b386-9d8d942bf929","Type":"ContainerStarted","Data":"6ddce84fdcc4323d0609d32909cfda9c0ff00e379e800778bdcff04edbb32653"}
Jul 29 23:18:54.032960 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.032716    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z" event={"ID":"54f76edb-55b1-4add-b386-9d8d942bf929","Type":"ContainerStarted","Data":"f669a0b817c333cd9b1ce1f93f5f8b64a91b29e63a26fecc62ef4ade8760b8a9"}
Jul 29 23:18:54.032960 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.032727    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z" event={"ID":"54f76edb-55b1-4add-b386-9d8d942bf929","Type":"ContainerStarted","Data":"120dc31bdc7d91c046d5e9749f10b7a3a9d59db5799c09005488d46060e8d82e"}
Jul 29 23:18:54.053387 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.053284    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f77ffc7d5-h8m9z" podStartSLOduration=1.053268668 podStartE2EDuration="1.053268668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:18:54.051900466 +0000 UTC m=+1941.057097890" watchObservedRunningTime="2026-07-29 23:18:54.053268668 +0000 UTC m=+1941.058466091"
Jul 29 23:18:54.088187 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.088155    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"]
Jul 29 23:18:54.088361 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.088343    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq" podUID="b21a2e37-ab9b-4e89-847f-cfbf19564217" containerName="proxy-agent" containerID="cri-o://bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007" gracePeriod=30
Jul 29 23:18:54.088409 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.088379    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq" podUID="b21a2e37-ab9b-4e89-847f-cfbf19564217" containerName="addon-agent" containerID="cri-o://3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581" gracePeriod=30
Jul 29 23:18:54.088537 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.088379    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq" podUID="b21a2e37-ab9b-4e89-847f-cfbf19564217" containerName="service-proxy" containerID="cri-o://7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f" gracePeriod=30
Jul 29 23:18:54.327173 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.327146    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:54.368162 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.368134    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-ca\") pod \"b21a2e37-ab9b-4e89-847f-cfbf19564217\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") "
Jul 29 23:18:54.368339 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.368176    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7tst\" (UniqueName: \"kubernetes.io/projected/b21a2e37-ab9b-4e89-847f-cfbf19564217-kube-api-access-b7tst\") pod \"b21a2e37-ab9b-4e89-847f-cfbf19564217\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") "
Jul 29 23:18:54.368339 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.368216    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-service-proxy-server-cert\") pod \"b21a2e37-ab9b-4e89-847f-cfbf19564217\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") "
Jul 29 23:18:54.368339 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.368232    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-hub-kubeconfig\") pod \"b21a2e37-ab9b-4e89-847f-cfbf19564217\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") "
Jul 29 23:18:54.368339 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.368282    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b21a2e37-ab9b-4e89-847f-cfbf19564217-additional-service-ca\") pod \"b21a2e37-ab9b-4e89-847f-cfbf19564217\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") "
Jul 29 23:18:54.368339 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.368300    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-hub\") pod \"b21a2e37-ab9b-4e89-847f-cfbf19564217\" (UID: \"b21a2e37-ab9b-4e89-847f-cfbf19564217\") "
Jul 29 23:18:54.369274 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.368960    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21a2e37-ab9b-4e89-847f-cfbf19564217-additional-service-ca" (OuterVolumeSpecName: "additional-service-ca") pod "b21a2e37-ab9b-4e89-847f-cfbf19564217" (UID: "b21a2e37-ab9b-4e89-847f-cfbf19564217"). InnerVolumeSpecName "additional-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 23:18:54.370825 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.370797    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-hub" (OuterVolumeSpecName: "hub") pod "b21a2e37-ab9b-4e89-847f-cfbf19564217" (UID: "b21a2e37-ab9b-4e89-847f-cfbf19564217"). InnerVolumeSpecName "hub". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:18:54.371390 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.370912    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-hub-kubeconfig" (OuterVolumeSpecName: "hub-kubeconfig") pod "b21a2e37-ab9b-4e89-847f-cfbf19564217" (UID: "b21a2e37-ab9b-4e89-847f-cfbf19564217"). InnerVolumeSpecName "hub-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:18:54.371390 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.370931    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-ca" (OuterVolumeSpecName: "ca") pod "b21a2e37-ab9b-4e89-847f-cfbf19564217" (UID: "b21a2e37-ab9b-4e89-847f-cfbf19564217"). InnerVolumeSpecName "ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:18:54.371390 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.371011    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21a2e37-ab9b-4e89-847f-cfbf19564217-kube-api-access-b7tst" (OuterVolumeSpecName: "kube-api-access-b7tst") pod "b21a2e37-ab9b-4e89-847f-cfbf19564217" (UID: "b21a2e37-ab9b-4e89-847f-cfbf19564217"). InnerVolumeSpecName "kube-api-access-b7tst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:18:54.371736 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.371710    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-service-proxy-server-cert" (OuterVolumeSpecName: "service-proxy-server-cert") pod "b21a2e37-ab9b-4e89-847f-cfbf19564217" (UID: "b21a2e37-ab9b-4e89-847f-cfbf19564217"). InnerVolumeSpecName "service-proxy-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:18:54.468932 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.468894    2569 reconciler_common.go:299] "Volume detached for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:18:54.468932 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.468925    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7tst\" (UniqueName: \"kubernetes.io/projected/b21a2e37-ab9b-4e89-847f-cfbf19564217-kube-api-access-b7tst\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:18:54.468932 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.468936    2569 reconciler_common.go:299] "Volume detached for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-service-proxy-server-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:18:54.469249 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.468947    2569 reconciler_common.go:299] "Volume detached for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-hub-kubeconfig\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:18:54.469249 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.468956    2569 reconciler_common.go:299] "Volume detached for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b21a2e37-ab9b-4e89-847f-cfbf19564217-additional-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:18:54.469249 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:54.468963    2569 reconciler_common.go:299] "Volume detached for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b21a2e37-ab9b-4e89-847f-cfbf19564217-hub\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:18:55.039001 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.038963    2569 generic.go:356] "Generic (PLEG): container finished" podID="b21a2e37-ab9b-4e89-847f-cfbf19564217" containerID="7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f" exitCode=2
Jul 29 23:18:55.039001 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.038989    2569 generic.go:356] "Generic (PLEG): container finished" podID="b21a2e37-ab9b-4e89-847f-cfbf19564217" containerID="3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581" exitCode=2
Jul 29 23:18:55.039001 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.038999    2569 generic.go:356] "Generic (PLEG): container finished" podID="b21a2e37-ab9b-4e89-847f-cfbf19564217" containerID="bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007" exitCode=2
Jul 29 23:18:55.039292 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.039017    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"
Jul 29 23:18:55.039292 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.039042    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq" event={"ID":"b21a2e37-ab9b-4e89-847f-cfbf19564217","Type":"ContainerDied","Data":"7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f"}
Jul 29 23:18:55.039292 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.039091    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq" event={"ID":"b21a2e37-ab9b-4e89-847f-cfbf19564217","Type":"ContainerDied","Data":"3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581"}
Jul 29 23:18:55.039292 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.039104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq" event={"ID":"b21a2e37-ab9b-4e89-847f-cfbf19564217","Type":"ContainerDied","Data":"bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007"}
Jul 29 23:18:55.039292 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.039116    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq" event={"ID":"b21a2e37-ab9b-4e89-847f-cfbf19564217","Type":"ContainerDied","Data":"e71592274810754d29e5012448e917711265b8162743c5cd8872af5027305005"}
Jul 29 23:18:55.039292 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.039132    2569 scope.go:117] "RemoveContainer" containerID="7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f"
Jul 29 23:18:55.048019 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.048000    2569 scope.go:117] "RemoveContainer" containerID="3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581"
Jul 29 23:18:55.055585 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.055566    2569 scope.go:117] "RemoveContainer" containerID="bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007"
Jul 29 23:18:55.062805 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.062787    2569 scope.go:117] "RemoveContainer" containerID="7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f"
Jul 29 23:18:55.063039 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:18:55.063020    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f\": container with ID starting with 7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f not found: ID does not exist" containerID="7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f"
Jul 29 23:18:55.063093 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.063048    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f"} err="failed to get container status \"7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f\": rpc error: code = NotFound desc = could not find container \"7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f\": container with ID starting with 7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f not found: ID does not exist"
Jul 29 23:18:55.063093 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.063074    2569 scope.go:117] "RemoveContainer" containerID="3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581"
Jul 29 23:18:55.063284 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:18:55.063270    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581\": container with ID starting with 3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581 not found: ID does not exist" containerID="3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581"
Jul 29 23:18:55.063326 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.063288    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581"} err="failed to get container status \"3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581\": rpc error: code = NotFound desc = could not find container \"3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581\": container with ID starting with 3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581 not found: ID does not exist"
Jul 29 23:18:55.063326 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.063300    2569 scope.go:117] "RemoveContainer" containerID="bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007"
Jul 29 23:18:55.063546 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:18:55.063522    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007\": container with ID starting with bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007 not found: ID does not exist" containerID="bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007"
Jul 29 23:18:55.063597 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.063556    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007"} err="failed to get container status \"bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007\": rpc error: code = NotFound desc = could not find container \"bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007\": container with ID starting with bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007 not found: ID does not exist"
Jul 29 23:18:55.063597 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.063577    2569 scope.go:117] "RemoveContainer" containerID="7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f"
Jul 29 23:18:55.063815 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.063789    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f"} err="failed to get container status \"7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f\": rpc error: code = NotFound desc = could not find container \"7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f\": container with ID starting with 7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f not found: ID does not exist"
Jul 29 23:18:55.063871 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.063815    2569 scope.go:117] "RemoveContainer" containerID="3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581"
Jul 29 23:18:55.064091 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.064063    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581"} err="failed to get container status \"3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581\": rpc error: code = NotFound desc = could not find container \"3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581\": container with ID starting with 3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581 not found: ID does not exist"
Jul 29 23:18:55.064175 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.064093    2569 scope.go:117] "RemoveContainer" containerID="bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007"
Jul 29 23:18:55.064504 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.064398    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007"} err="failed to get container status \"bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007\": rpc error: code = NotFound desc = could not find container \"bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007\": container with ID starting with bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007 not found: ID does not exist"
Jul 29 23:18:55.064504 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.064424    2569 scope.go:117] "RemoveContainer" containerID="7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f"
Jul 29 23:18:55.064734 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.064692    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f"} err="failed to get container status \"7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f\": rpc error: code = NotFound desc = could not find container \"7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f\": container with ID starting with 7f28c2dc5edb1f76da1fa975c44ecb0f28749de0a8f4917c1e7954bb575a2c9f not found: ID does not exist"
Jul 29 23:18:55.064843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.064736    2569 scope.go:117] "RemoveContainer" containerID="3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581"
Jul 29 23:18:55.065048 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.065028    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581"} err="failed to get container status \"3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581\": rpc error: code = NotFound desc = could not find container \"3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581\": container with ID starting with 3fc3258f46b1b7926a0515558b4444cf6887b07b0f30fc53252a7715899ca581 not found: ID does not exist"
Jul 29 23:18:55.065125 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.065049    2569 scope.go:117] "RemoveContainer" containerID="bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007"
Jul 29 23:18:55.065287 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.065263    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007"} err="failed to get container status \"bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007\": rpc error: code = NotFound desc = could not find container \"bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007\": container with ID starting with bbc681a8040aa4396a412bbc5a3736d81f5753eb5f56fbce817c82017129a007 not found: ID does not exist"
Jul 29 23:18:55.066636 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.066618    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"]
Jul 29 23:18:55.068163 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.068141    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b8ffbb94f-zf7sq"]
Jul 29 23:18:55.474023 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:18:55.473989    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b21a2e37-ab9b-4e89-847f-cfbf19564217" path="/var/lib/kubelet/pods/b21a2e37-ab9b-4e89-847f-cfbf19564217/volumes"
Jul 29 23:19:21.817440 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:21.817399    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"]
Jul 29 23:19:21.827519 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:19:21.827462    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:19:21.827519 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:19:21.827519 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:19:21.827519 ip-10-0-135-98 kubenswrapper[2569]:  > containerID="9381ee58acc92eb2d8c611725d46ac637f0a4b43cc407d962901d72f4e22c514" cmd=["/bin/sleep","15"]
Jul 29 23:19:21.827836 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:21.827546    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd" message="" err=<
Jul 29 23:19:21.827836 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:19:21.827836 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:19:21.827836 ip-10-0-135-98 kubenswrapper[2569]:  >
Jul 29 23:19:21.827836 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:19:21.827566    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:19:21.827836 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:19:21.827836 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:19:21.827836 ip-10-0-135-98 kubenswrapper[2569]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd" podUID="6187990c-b60e-4602-b176-46676bab619d" containerName="main" containerID="cri-o://9381ee58acc92eb2d8c611725d46ac637f0a4b43cc407d962901d72f4e22c514"
Jul 29 23:19:21.827836 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:21.827626    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd" podUID="6187990c-b60e-4602-b176-46676bab619d" containerName="main" containerID="cri-o://9381ee58acc92eb2d8c611725d46ac637f0a4b43cc407d962901d72f4e22c514" gracePeriod=60
Jul 29 23:19:21.871171 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:21.871135    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd" podUID="6187990c-b60e-4602-b176-46676bab619d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 29 23:19:22.152506 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.152450    2569 generic.go:356] "Generic (PLEG): container finished" podID="6187990c-b60e-4602-b176-46676bab619d" containerID="9381ee58acc92eb2d8c611725d46ac637f0a4b43cc407d962901d72f4e22c514" exitCode=0
Jul 29 23:19:22.152678 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.152524    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd" event={"ID":"6187990c-b60e-4602-b176-46676bab619d","Type":"ContainerDied","Data":"9381ee58acc92eb2d8c611725d46ac637f0a4b43cc407d962901d72f4e22c514"}
Jul 29 23:19:22.180728 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.180698    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:19:22.319759 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.319723    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-dshm\") pod \"6187990c-b60e-4602-b176-46676bab619d\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") "
Jul 29 23:19:22.319759 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.319766    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-home\") pod \"6187990c-b60e-4602-b176-46676bab619d\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") "
Jul 29 23:19:22.320037 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.319794    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-model-cache\") pod \"6187990c-b60e-4602-b176-46676bab619d\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") "
Jul 29 23:19:22.320037 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.319857    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6187990c-b60e-4602-b176-46676bab619d-tls-certs\") pod \"6187990c-b60e-4602-b176-46676bab619d\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") "
Jul 29 23:19:22.320037 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.319882    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtvtc\" (UniqueName: \"kubernetes.io/projected/6187990c-b60e-4602-b176-46676bab619d-kube-api-access-rtvtc\") pod \"6187990c-b60e-4602-b176-46676bab619d\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") "
Jul 29 23:19:22.320037 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.319904    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-tmp-dir\") pod \"6187990c-b60e-4602-b176-46676bab619d\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") "
Jul 29 23:19:22.320261 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.320050    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-kserve-provision-location\") pod \"6187990c-b60e-4602-b176-46676bab619d\" (UID: \"6187990c-b60e-4602-b176-46676bab619d\") "
Jul 29 23:19:22.320261 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.320061    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-home" (OuterVolumeSpecName: "home") pod "6187990c-b60e-4602-b176-46676bab619d" (UID: "6187990c-b60e-4602-b176-46676bab619d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:19:22.320261 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.320241    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6187990c-b60e-4602-b176-46676bab619d" (UID: "6187990c-b60e-4602-b176-46676bab619d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:19:22.320422 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.320345    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.320422 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.320371    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.320554 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.320511    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6187990c-b60e-4602-b176-46676bab619d" (UID: "6187990c-b60e-4602-b176-46676bab619d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:19:22.322322 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.322292    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-dshm" (OuterVolumeSpecName: "dshm") pod "6187990c-b60e-4602-b176-46676bab619d" (UID: "6187990c-b60e-4602-b176-46676bab619d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:19:22.322322 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.322316    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6187990c-b60e-4602-b176-46676bab619d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6187990c-b60e-4602-b176-46676bab619d" (UID: "6187990c-b60e-4602-b176-46676bab619d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:19:22.322472 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.322449    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6187990c-b60e-4602-b176-46676bab619d-kube-api-access-rtvtc" (OuterVolumeSpecName: "kube-api-access-rtvtc") pod "6187990c-b60e-4602-b176-46676bab619d" (UID: "6187990c-b60e-4602-b176-46676bab619d"). InnerVolumeSpecName "kube-api-access-rtvtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:19:22.378884 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.378790    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6187990c-b60e-4602-b176-46676bab619d" (UID: "6187990c-b60e-4602-b176-46676bab619d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:19:22.421301 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.421269    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6187990c-b60e-4602-b176-46676bab619d-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.421301 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.421297    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtvtc\" (UniqueName: \"kubernetes.io/projected/6187990c-b60e-4602-b176-46676bab619d-kube-api-access-rtvtc\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.421301 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.421307    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-kserve-provision-location\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.421555 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.421316    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.421555 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:22.421324    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6187990c-b60e-4602-b176-46676bab619d-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:19:23.158363 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:23.158326    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"
Jul 29 23:19:23.158889 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:23.158324    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd" event={"ID":"6187990c-b60e-4602-b176-46676bab619d","Type":"ContainerDied","Data":"80c7e7f4252813012934eca85b247cc7aab824f8f2ac8849cef07b5517f179c1"}
Jul 29 23:19:23.158889 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:23.158514    2569 scope.go:117] "RemoveContainer" containerID="9381ee58acc92eb2d8c611725d46ac637f0a4b43cc407d962901d72f4e22c514"
Jul 29 23:19:23.168021 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:23.168001    2569 scope.go:117] "RemoveContainer" containerID="7250ac3acad65e0243ea2dc64f9d07bce7794c42a7272ce8ad0c74d6f8b2eda1"
Jul 29 23:19:23.178860 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:23.178835    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"]
Jul 29 23:19:23.182970 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:23.182947    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-tj6jd"]
Jul 29 23:19:23.474031 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:23.474000    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6187990c-b60e-4602-b176-46676bab619d" path="/var/lib/kubelet/pods/6187990c-b60e-4602-b176-46676bab619d/volumes"
Jul 29 23:19:54.797632 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.797596    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"]
Jul 29 23:19:54.798192 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.798044    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21a2e37-ab9b-4e89-847f-cfbf19564217" containerName="proxy-agent"
Jul 29 23:19:54.798192 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.798064    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21a2e37-ab9b-4e89-847f-cfbf19564217" containerName="proxy-agent"
Jul 29 23:19:54.798192 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.798079    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6187990c-b60e-4602-b176-46676bab619d" containerName="storage-initializer"
Jul 29 23:19:54.798192 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.798089    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6187990c-b60e-4602-b176-46676bab619d" containerName="storage-initializer"
Jul 29 23:19:54.798192 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.798104    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21a2e37-ab9b-4e89-847f-cfbf19564217" containerName="addon-agent"
Jul 29 23:19:54.798192 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.798112    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21a2e37-ab9b-4e89-847f-cfbf19564217" containerName="addon-agent"
Jul 29 23:19:54.798192 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.798127    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21a2e37-ab9b-4e89-847f-cfbf19564217" containerName="service-proxy"
Jul 29 23:19:54.798192 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.798135    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21a2e37-ab9b-4e89-847f-cfbf19564217" containerName="service-proxy"
Jul 29 23:19:54.798192 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.798145    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6187990c-b60e-4602-b176-46676bab619d" containerName="main"
Jul 29 23:19:54.798192 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.798153    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6187990c-b60e-4602-b176-46676bab619d" containerName="main"
Jul 29 23:19:54.798764 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.798262    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6187990c-b60e-4602-b176-46676bab619d" containerName="main"
Jul 29 23:19:54.798764 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.798274    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b21a2e37-ab9b-4e89-847f-cfbf19564217" containerName="service-proxy"
Jul 29 23:19:54.798764 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.798285    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b21a2e37-ab9b-4e89-847f-cfbf19564217" containerName="addon-agent"
Jul 29 23:19:54.798764 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.798297    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b21a2e37-ab9b-4e89-847f-cfbf19564217" containerName="proxy-agent"
Jul 29 23:19:54.801795 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.801771    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.804524 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.804497    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-9lq75\""
Jul 29 23:19:54.804657 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.804566    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 29 23:19:54.804657 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.804502    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 29 23:19:54.804657 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.804649    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 29 23:19:54.804855 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.804724    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 29 23:19:54.813707 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.813675    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"]
Jul 29 23:19:54.891445 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.891409    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.891653 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.891465    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-model-cache\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.891653 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.891558    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6443c76-54fe-45e9-91b6-b18898dd9693-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.891653 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.891597    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tkbj\" (UniqueName: \"kubernetes.io/projected/e6443c76-54fe-45e9-91b6-b18898dd9693-kube-api-access-9tkbj\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.891823 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.891673    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.891823 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.891702    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-home\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.891823 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.891736    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-dshm\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.992727 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.992698    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-model-cache\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.992727 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.992737    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6443c76-54fe-45e9-91b6-b18898dd9693-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.992981 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.992763    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tkbj\" (UniqueName: \"kubernetes.io/projected/e6443c76-54fe-45e9-91b6-b18898dd9693-kube-api-access-9tkbj\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.992981 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.992811    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.992981 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.992838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-home\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.992981 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.992868    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-dshm\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.992981 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.992915    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.993270 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.993123    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-model-cache\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.993270 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.993193    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-home\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.993270 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.993250    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.993399 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.993322    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.995051 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.995034    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-dshm\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:54.995457 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:54.995437    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6443c76-54fe-45e9-91b6-b18898dd9693-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:55.001568 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:55.001543    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tkbj\" (UniqueName: \"kubernetes.io/projected/e6443c76-54fe-45e9-91b6-b18898dd9693-kube-api-access-9tkbj\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-pv75l\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:55.115657 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:55.115560    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:19:55.250136 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:55.250040    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"]
Jul 29 23:19:55.252742 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:19:55.252713    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6443c76_54fe_45e9_91b6_b18898dd9693.slice/crio-2bdf86919449dc1ed8c326a90627a2d36aa11beb6ae01efc926eae4cb017890d WatchSource:0}: Error finding container 2bdf86919449dc1ed8c326a90627a2d36aa11beb6ae01efc926eae4cb017890d: Status 404 returned error can't find the container with id 2bdf86919449dc1ed8c326a90627a2d36aa11beb6ae01efc926eae4cb017890d
Jul 29 23:19:55.295246 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:55.295217    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l" event={"ID":"e6443c76-54fe-45e9-91b6-b18898dd9693","Type":"ContainerStarted","Data":"2bdf86919449dc1ed8c326a90627a2d36aa11beb6ae01efc926eae4cb017890d"}
Jul 29 23:19:56.303814 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:19:56.303776    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l" event={"ID":"e6443c76-54fe-45e9-91b6-b18898dd9693","Type":"ContainerStarted","Data":"42e12f5e866e2202c436a5765aea376848556d667fc99efd35b98d159ff5214d"}
Jul 29 23:20:00.324584 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:00.324544    2569 generic.go:356] "Generic (PLEG): container finished" podID="e6443c76-54fe-45e9-91b6-b18898dd9693" containerID="42e12f5e866e2202c436a5765aea376848556d667fc99efd35b98d159ff5214d" exitCode=0
Jul 29 23:20:00.325015 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:00.324612    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l" event={"ID":"e6443c76-54fe-45e9-91b6-b18898dd9693","Type":"ContainerDied","Data":"42e12f5e866e2202c436a5765aea376848556d667fc99efd35b98d159ff5214d"}
Jul 29 23:20:01.330767 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:01.330730    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l" event={"ID":"e6443c76-54fe-45e9-91b6-b18898dd9693","Type":"ContainerStarted","Data":"870f1861d31562b3eb24441567c9b8bcad579953049dc1859c96356cfdb64caf"}
Jul 29 23:20:01.349181 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:01.349125    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l" podStartSLOduration=2.8765638879999997 podStartE2EDuration="7.349109307s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.472545419s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:20:01.34797392 +0000 UTC m=+2008.353171342" watchObservedRunningTime="2026-07-29 23:20:01.349109307 +0000 UTC m=+2008.354306729"
Jul 29 23:20:02.116686 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:02.116641    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:20:05.115878 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:05.115833    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:20:05.128467 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:05.128443    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:20:05.358988 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:05.358960    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:20:33.951803 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:33.951766    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"]
Jul 29 23:20:33.962646 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:20:33.962605    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:20:33.962646 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:20:33.962646 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:20:33.962646 ip-10-0-135-98 kubenswrapper[2569]:  > containerID="870f1861d31562b3eb24441567c9b8bcad579953049dc1859c96356cfdb64caf" cmd=["/bin/sleep","15"]
Jul 29 23:20:33.962979 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:33.962684    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l" message="" err=<
Jul 29 23:20:33.962979 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:20:33.962979 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:20:33.962979 ip-10-0-135-98 kubenswrapper[2569]:  >
Jul 29 23:20:33.962979 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:20:33.962702    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:20:33.962979 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:20:33.962979 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:20:33.962979 ip-10-0-135-98 kubenswrapper[2569]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l" podUID="e6443c76-54fe-45e9-91b6-b18898dd9693" containerName="main" containerID="cri-o://870f1861d31562b3eb24441567c9b8bcad579953049dc1859c96356cfdb64caf"
Jul 29 23:20:33.962979 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:33.962747    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l" podUID="e6443c76-54fe-45e9-91b6-b18898dd9693" containerName="main" containerID="cri-o://870f1861d31562b3eb24441567c9b8bcad579953049dc1859c96356cfdb64caf" gracePeriod=60
Jul 29 23:20:34.303539 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.303508    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:20:34.346176 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.346146    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6443c76-54fe-45e9-91b6-b18898dd9693-tls-certs\") pod \"e6443c76-54fe-45e9-91b6-b18898dd9693\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") "
Jul 29 23:20:34.346384 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.346198    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-dshm\") pod \"e6443c76-54fe-45e9-91b6-b18898dd9693\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") "
Jul 29 23:20:34.346384 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.346228    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-kserve-provision-location\") pod \"e6443c76-54fe-45e9-91b6-b18898dd9693\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") "
Jul 29 23:20:34.346384 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.346273    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tkbj\" (UniqueName: \"kubernetes.io/projected/e6443c76-54fe-45e9-91b6-b18898dd9693-kube-api-access-9tkbj\") pod \"e6443c76-54fe-45e9-91b6-b18898dd9693\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") "
Jul 29 23:20:34.346384 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.346298    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-model-cache\") pod \"e6443c76-54fe-45e9-91b6-b18898dd9693\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") "
Jul 29 23:20:34.346384 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.346343    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-home\") pod \"e6443c76-54fe-45e9-91b6-b18898dd9693\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") "
Jul 29 23:20:34.346675 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.346400    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-tmp-dir\") pod \"e6443c76-54fe-45e9-91b6-b18898dd9693\" (UID: \"e6443c76-54fe-45e9-91b6-b18898dd9693\") "
Jul 29 23:20:34.346917 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.346749    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6443c76-54fe-45e9-91b6-b18898dd9693" (UID: "e6443c76-54fe-45e9-91b6-b18898dd9693"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:20:34.346917 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.346863    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6443c76-54fe-45e9-91b6-b18898dd9693" (UID: "e6443c76-54fe-45e9-91b6-b18898dd9693"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:20:34.346917 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.346898    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-home" (OuterVolumeSpecName: "home") pod "e6443c76-54fe-45e9-91b6-b18898dd9693" (UID: "e6443c76-54fe-45e9-91b6-b18898dd9693"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:20:34.349066 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.349034    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6443c76-54fe-45e9-91b6-b18898dd9693-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6443c76-54fe-45e9-91b6-b18898dd9693" (UID: "e6443c76-54fe-45e9-91b6-b18898dd9693"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:20:34.349173 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.349034    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-dshm" (OuterVolumeSpecName: "dshm") pod "e6443c76-54fe-45e9-91b6-b18898dd9693" (UID: "e6443c76-54fe-45e9-91b6-b18898dd9693"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:20:34.349173 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.349140    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6443c76-54fe-45e9-91b6-b18898dd9693-kube-api-access-9tkbj" (OuterVolumeSpecName: "kube-api-access-9tkbj") pod "e6443c76-54fe-45e9-91b6-b18898dd9693" (UID: "e6443c76-54fe-45e9-91b6-b18898dd9693"). InnerVolumeSpecName "kube-api-access-9tkbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:20:34.404613 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.404556    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6443c76-54fe-45e9-91b6-b18898dd9693" (UID: "e6443c76-54fe-45e9-91b6-b18898dd9693"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:20:34.447007 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.446971    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tkbj\" (UniqueName: \"kubernetes.io/projected/e6443c76-54fe-45e9-91b6-b18898dd9693-kube-api-access-9tkbj\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.447007 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.447003    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.447007 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.447013    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.447256 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.447044    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.447256 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.447052    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6443c76-54fe-45e9-91b6-b18898dd9693-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.447256 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.447059    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.447256 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.447067    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6443c76-54fe-45e9-91b6-b18898dd9693-kserve-provision-location\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.467888 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.467854    2569 generic.go:356] "Generic (PLEG): container finished" podID="e6443c76-54fe-45e9-91b6-b18898dd9693" containerID="870f1861d31562b3eb24441567c9b8bcad579953049dc1859c96356cfdb64caf" exitCode=0
Jul 29 23:20:34.468063 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.467909    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"
Jul 29 23:20:34.468063 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.467906    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l" event={"ID":"e6443c76-54fe-45e9-91b6-b18898dd9693","Type":"ContainerDied","Data":"870f1861d31562b3eb24441567c9b8bcad579953049dc1859c96356cfdb64caf"}
Jul 29 23:20:34.468063 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.468031    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l" event={"ID":"e6443c76-54fe-45e9-91b6-b18898dd9693","Type":"ContainerDied","Data":"2bdf86919449dc1ed8c326a90627a2d36aa11beb6ae01efc926eae4cb017890d"}
Jul 29 23:20:34.468063 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.468061    2569 scope.go:117] "RemoveContainer" containerID="870f1861d31562b3eb24441567c9b8bcad579953049dc1859c96356cfdb64caf"
Jul 29 23:20:34.477187 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.477104    2569 scope.go:117] "RemoveContainer" containerID="42e12f5e866e2202c436a5765aea376848556d667fc99efd35b98d159ff5214d"
Jul 29 23:20:34.485389 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.485363    2569 scope.go:117] "RemoveContainer" containerID="870f1861d31562b3eb24441567c9b8bcad579953049dc1859c96356cfdb64caf"
Jul 29 23:20:34.485700 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:20:34.485681    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870f1861d31562b3eb24441567c9b8bcad579953049dc1859c96356cfdb64caf\": container with ID starting with 870f1861d31562b3eb24441567c9b8bcad579953049dc1859c96356cfdb64caf not found: ID does not exist" containerID="870f1861d31562b3eb24441567c9b8bcad579953049dc1859c96356cfdb64caf"
Jul 29 23:20:34.485766 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.485711    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870f1861d31562b3eb24441567c9b8bcad579953049dc1859c96356cfdb64caf"} err="failed to get container status \"870f1861d31562b3eb24441567c9b8bcad579953049dc1859c96356cfdb64caf\": rpc error: code = NotFound desc = could not find container \"870f1861d31562b3eb24441567c9b8bcad579953049dc1859c96356cfdb64caf\": container with ID starting with 870f1861d31562b3eb24441567c9b8bcad579953049dc1859c96356cfdb64caf not found: ID does not exist"
Jul 29 23:20:34.485766 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.485730    2569 scope.go:117] "RemoveContainer" containerID="42e12f5e866e2202c436a5765aea376848556d667fc99efd35b98d159ff5214d"
Jul 29 23:20:34.485938 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:20:34.485921    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e12f5e866e2202c436a5765aea376848556d667fc99efd35b98d159ff5214d\": container with ID starting with 42e12f5e866e2202c436a5765aea376848556d667fc99efd35b98d159ff5214d not found: ID does not exist" containerID="42e12f5e866e2202c436a5765aea376848556d667fc99efd35b98d159ff5214d"
Jul 29 23:20:34.485983 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.485944    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e12f5e866e2202c436a5765aea376848556d667fc99efd35b98d159ff5214d"} err="failed to get container status \"42e12f5e866e2202c436a5765aea376848556d667fc99efd35b98d159ff5214d\": rpc error: code = NotFound desc = could not find container \"42e12f5e866e2202c436a5765aea376848556d667fc99efd35b98d159ff5214d\": container with ID starting with 42e12f5e866e2202c436a5765aea376848556d667fc99efd35b98d159ff5214d not found: ID does not exist"
Jul 29 23:20:34.490703 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.490678    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"]
Jul 29 23:20:34.496811 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:34.496787    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-pv75l"]
Jul 29 23:20:35.474311 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:20:35.474275    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6443c76-54fe-45e9-91b6-b18898dd9693" path="/var/lib/kubelet/pods/e6443c76-54fe-45e9-91b6-b18898dd9693/volumes"
Jul 29 23:21:33.573607 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:21:33.573530    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:21:33.581140 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:21:33.581115    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:22:41.390757 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.390717    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"]
Jul 29 23:22:41.392293 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.391001    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6443c76-54fe-45e9-91b6-b18898dd9693" containerName="main"
Jul 29 23:22:41.392293 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.391012    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6443c76-54fe-45e9-91b6-b18898dd9693" containerName="main"
Jul 29 23:22:41.392293 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.391038    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6443c76-54fe-45e9-91b6-b18898dd9693" containerName="storage-initializer"
Jul 29 23:22:41.392293 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.391043    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6443c76-54fe-45e9-91b6-b18898dd9693" containerName="storage-initializer"
Jul 29 23:22:41.392293 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.391104    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6443c76-54fe-45e9-91b6-b18898dd9693" containerName="main"
Jul 29 23:22:41.392736 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.392719    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.395333 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.395306    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-56mjh\""
Jul 29 23:22:41.395333 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.395325    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 29 23:22:41.395531 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.395313    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 29 23:22:41.395531 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.395327    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 29 23:22:41.402542 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.402517    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"]
Jul 29 23:22:41.504122 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.504089    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.504315 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.504184    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.504315 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.504208    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba005bb-2e30-4c84-9814-2a079c993446-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.504315 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.504226    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.504315 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.504241    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.504476 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.504360    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnn4w\" (UniqueName: \"kubernetes.io/projected/aba005bb-2e30-4c84-9814-2a079c993446-kube-api-access-gnn4w\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.580778 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.580740    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"]
Jul 29 23:22:41.582827 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.582803    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"
Jul 29 23:22:41.585220 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.585194    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-nwbpp\""
Jul 29 23:22:41.595606 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.595582    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"]
Jul 29 23:22:41.605068 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.605038    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnn4w\" (UniqueName: \"kubernetes.io/projected/aba005bb-2e30-4c84-9814-2a079c993446-kube-api-access-gnn4w\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.605232 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.605076    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.605232 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.605112    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ac6824-07be-4095-82ce-47dfe0ca8098-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb\" (UID: \"c4ac6824-07be-4095-82ce-47dfe0ca8098\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"
Jul 29 23:22:41.605358 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.605238    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldg9j\" (UniqueName: \"kubernetes.io/projected/c4ac6824-07be-4095-82ce-47dfe0ca8098-kube-api-access-ldg9j\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb\" (UID: \"c4ac6824-07be-4095-82ce-47dfe0ca8098\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"
Jul 29 23:22:41.605358 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.605326    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.605475 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.605369    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba005bb-2e30-4c84-9814-2a079c993446-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.605475 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.605401    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.605475 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.605427    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.605641 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.605547    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.605746 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.605724    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.605862 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.605835    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.608029 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.608005    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.608174 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.608151    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba005bb-2e30-4c84-9814-2a079c993446-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.616581 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.616529    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnn4w\" (UniqueName: \"kubernetes.io/projected/aba005bb-2e30-4c84-9814-2a079c993446-kube-api-access-gnn4w\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.705229 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.705140    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:41.706433 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.706404    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ac6824-07be-4095-82ce-47dfe0ca8098-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb\" (UID: \"c4ac6824-07be-4095-82ce-47dfe0ca8098\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"
Jul 29 23:22:41.706516 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.706446    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldg9j\" (UniqueName: \"kubernetes.io/projected/c4ac6824-07be-4095-82ce-47dfe0ca8098-kube-api-access-ldg9j\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb\" (UID: \"c4ac6824-07be-4095-82ce-47dfe0ca8098\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"
Jul 29 23:22:41.709253 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.709226    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ac6824-07be-4095-82ce-47dfe0ca8098-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb\" (UID: \"c4ac6824-07be-4095-82ce-47dfe0ca8098\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"
Jul 29 23:22:41.714619 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.714518    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldg9j\" (UniqueName: \"kubernetes.io/projected/c4ac6824-07be-4095-82ce-47dfe0ca8098-kube-api-access-ldg9j\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb\" (UID: \"c4ac6824-07be-4095-82ce-47dfe0ca8098\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"
Jul 29 23:22:41.839463 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.839432    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"]
Jul 29 23:22:41.842583 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:22:41.842554    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba005bb_2e30_4c84_9814_2a079c993446.slice/crio-d5d329878106109c3155bf5828f8bf88ee651fb6856651377dd7153cf75cf9e3 WatchSource:0}: Error finding container d5d329878106109c3155bf5828f8bf88ee651fb6856651377dd7153cf75cf9e3: Status 404 returned error can't find the container with id d5d329878106109c3155bf5828f8bf88ee651fb6856651377dd7153cf75cf9e3
Jul 29 23:22:41.893051 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.893027    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"
Jul 29 23:22:41.980113 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.980076    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9" event={"ID":"aba005bb-2e30-4c84-9814-2a079c993446","Type":"ContainerStarted","Data":"553d164c1c57534c4a384131746b59b328717acd2b37fd496ba7d373bed864da"}
Jul 29 23:22:41.980257 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.980143    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9" event={"ID":"aba005bb-2e30-4c84-9814-2a079c993446","Type":"ContainerStarted","Data":"d5d329878106109c3155bf5828f8bf88ee651fb6856651377dd7153cf75cf9e3"}
Jul 29 23:22:41.998226 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:41.998178    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9" podStartSLOduration=0.998161728 podStartE2EDuration="998.161728ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:22:41.99670355 +0000 UTC m=+2169.001900984" watchObservedRunningTime="2026-07-29 23:22:41.998161728 +0000 UTC m=+2169.003359151"
Jul 29 23:22:42.030167 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:42.030104    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"]
Jul 29 23:22:42.032645 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:22:42.032613    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ac6824_07be_4095_82ce_47dfe0ca8098.slice/crio-25fd43a7d3179bb11e85ff228a94b5b04f36ac4a5cd73d2b31dec5375e297e2e WatchSource:0}: Error finding container 25fd43a7d3179bb11e85ff228a94b5b04f36ac4a5cd73d2b31dec5375e297e2e: Status 404 returned error can't find the container with id 25fd43a7d3179bb11e85ff228a94b5b04f36ac4a5cd73d2b31dec5375e297e2e
Jul 29 23:22:42.706113 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:42.706077    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:42.986012 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:42.985982    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb" event={"ID":"c4ac6824-07be-4095-82ce-47dfe0ca8098","Type":"ContainerStarted","Data":"027a9c20f45608bfd7ac4b66df09344e1c7701f9961945a90d1ec52f9055deda"}
Jul 29 23:22:42.986228 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:42.986032    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb" event={"ID":"c4ac6824-07be-4095-82ce-47dfe0ca8098","Type":"ContainerStarted","Data":"25fd43a7d3179bb11e85ff228a94b5b04f36ac4a5cd73d2b31dec5375e297e2e"}
Jul 29 23:22:42.986228 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:42.986086    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"
Jul 29 23:22:43.007810 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:43.007751    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb" podStartSLOduration=2.007735359 podStartE2EDuration="2.007735359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:22:43.005463088 +0000 UTC m=+2170.010660516" watchObservedRunningTime="2026-07-29 23:22:43.007735359 +0000 UTC m=+2170.012932782"
Jul 29 23:22:51.706099 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:51.706058    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:51.718716 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:51.718687    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:22:52.041670 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:22:52.041642    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:23:13.992401 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:13.992371    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"
Jul 29 23:23:15.102257 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.102216    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"]
Jul 29 23:23:15.107345 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.107318    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"]
Jul 29 23:23:15.113985 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:23:15.113943    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:23:15.113985 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:23:15.113985 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:23:15.113985 ip-10-0-135-98 kubenswrapper[2569]:  > containerID="553d164c1c57534c4a384131746b59b328717acd2b37fd496ba7d373bed864da" cmd=["/bin/sleep","15"]
Jul 29 23:23:15.114283 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.113992    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9" message="" err=<
Jul 29 23:23:15.114283 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:23:15.114283 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:23:15.114283 ip-10-0-135-98 kubenswrapper[2569]:  >
Jul 29 23:23:15.114283 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:23:15.114006    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:23:15.114283 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:23:15.114283 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:23:15.114283 ip-10-0-135-98 kubenswrapper[2569]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9" podUID="aba005bb-2e30-4c84-9814-2a079c993446" containerName="main" containerID="cri-o://553d164c1c57534c4a384131746b59b328717acd2b37fd496ba7d373bed864da"
Jul 29 23:23:15.114283 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.114041    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9" podUID="aba005bb-2e30-4c84-9814-2a079c993446" containerName="main" containerID="cri-o://553d164c1c57534c4a384131746b59b328717acd2b37fd496ba7d373bed864da" gracePeriod=60
Jul 29 23:23:15.376656 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.376628    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:23:15.496896 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.496855    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-tmp-dir\") pod \"aba005bb-2e30-4c84-9814-2a079c993446\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") "
Jul 29 23:23:15.497083 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.496946    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba005bb-2e30-4c84-9814-2a079c993446-tls-certs\") pod \"aba005bb-2e30-4c84-9814-2a079c993446\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") "
Jul 29 23:23:15.497083 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.496972    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-dshm\") pod \"aba005bb-2e30-4c84-9814-2a079c993446\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") "
Jul 29 23:23:15.497083 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.496994    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnn4w\" (UniqueName: \"kubernetes.io/projected/aba005bb-2e30-4c84-9814-2a079c993446-kube-api-access-gnn4w\") pod \"aba005bb-2e30-4c84-9814-2a079c993446\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") "
Jul 29 23:23:15.497281 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.497158    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aba005bb-2e30-4c84-9814-2a079c993446" (UID: "aba005bb-2e30-4c84-9814-2a079c993446"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:23:15.497281 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.497184    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-model-cache\") pod \"aba005bb-2e30-4c84-9814-2a079c993446\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") "
Jul 29 23:23:15.497281 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.497258    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-home\") pod \"aba005bb-2e30-4c84-9814-2a079c993446\" (UID: \"aba005bb-2e30-4c84-9814-2a079c993446\") "
Jul 29 23:23:15.497417 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.497391    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-model-cache" (OuterVolumeSpecName: "model-cache") pod "aba005bb-2e30-4c84-9814-2a079c993446" (UID: "aba005bb-2e30-4c84-9814-2a079c993446"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:23:15.497565 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.497541    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-home" (OuterVolumeSpecName: "home") pod "aba005bb-2e30-4c84-9814-2a079c993446" (UID: "aba005bb-2e30-4c84-9814-2a079c993446"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:23:15.497627 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.497556    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:23:15.497627 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.497592    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:23:15.499204 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.499182    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-dshm" (OuterVolumeSpecName: "dshm") pod "aba005bb-2e30-4c84-9814-2a079c993446" (UID: "aba005bb-2e30-4c84-9814-2a079c993446"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:23:15.499560 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.499540    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba005bb-2e30-4c84-9814-2a079c993446-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aba005bb-2e30-4c84-9814-2a079c993446" (UID: "aba005bb-2e30-4c84-9814-2a079c993446"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:23:15.499560 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.499551    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba005bb-2e30-4c84-9814-2a079c993446-kube-api-access-gnn4w" (OuterVolumeSpecName: "kube-api-access-gnn4w") pod "aba005bb-2e30-4c84-9814-2a079c993446" (UID: "aba005bb-2e30-4c84-9814-2a079c993446"). InnerVolumeSpecName "kube-api-access-gnn4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:23:15.598566 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.598530    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:23:15.598566 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.598562    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba005bb-2e30-4c84-9814-2a079c993446-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:23:15.598566 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.598574    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba005bb-2e30-4c84-9814-2a079c993446-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:23:15.598798 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:15.598583    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnn4w\" (UniqueName: \"kubernetes.io/projected/aba005bb-2e30-4c84-9814-2a079c993446-kube-api-access-gnn4w\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:23:16.134260 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:16.134227    2569 generic.go:356] "Generic (PLEG): container finished" podID="aba005bb-2e30-4c84-9814-2a079c993446" containerID="553d164c1c57534c4a384131746b59b328717acd2b37fd496ba7d373bed864da" exitCode=0
Jul 29 23:23:16.134760 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:16.134273    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"
Jul 29 23:23:16.134760 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:16.134313    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9" event={"ID":"aba005bb-2e30-4c84-9814-2a079c993446","Type":"ContainerDied","Data":"553d164c1c57534c4a384131746b59b328717acd2b37fd496ba7d373bed864da"}
Jul 29 23:23:16.134760 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:16.134372    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9" event={"ID":"aba005bb-2e30-4c84-9814-2a079c993446","Type":"ContainerDied","Data":"d5d329878106109c3155bf5828f8bf88ee651fb6856651377dd7153cf75cf9e3"}
Jul 29 23:23:16.134760 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:16.134410    2569 scope.go:117] "RemoveContainer" containerID="553d164c1c57534c4a384131746b59b328717acd2b37fd496ba7d373bed864da"
Jul 29 23:23:16.145638 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:16.145607    2569 scope.go:117] "RemoveContainer" containerID="553d164c1c57534c4a384131746b59b328717acd2b37fd496ba7d373bed864da"
Jul 29 23:23:16.146255 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:23:16.146231    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553d164c1c57534c4a384131746b59b328717acd2b37fd496ba7d373bed864da\": container with ID starting with 553d164c1c57534c4a384131746b59b328717acd2b37fd496ba7d373bed864da not found: ID does not exist" containerID="553d164c1c57534c4a384131746b59b328717acd2b37fd496ba7d373bed864da"
Jul 29 23:23:16.146353 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:16.146265    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553d164c1c57534c4a384131746b59b328717acd2b37fd496ba7d373bed864da"} err="failed to get container status \"553d164c1c57534c4a384131746b59b328717acd2b37fd496ba7d373bed864da\": rpc error: code = NotFound desc = could not find container \"553d164c1c57534c4a384131746b59b328717acd2b37fd496ba7d373bed864da\": container with ID starting with 553d164c1c57534c4a384131746b59b328717acd2b37fd496ba7d373bed864da not found: ID does not exist"
Jul 29 23:23:16.159065 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:16.159030    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"]
Jul 29 23:23:16.162522 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:16.162497    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995zmvc9"]
Jul 29 23:23:17.474440 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:17.474410    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aba005bb-2e30-4c84-9814-2a079c993446" path="/var/lib/kubelet/pods/aba005bb-2e30-4c84-9814-2a079c993446/volumes"
Jul 29 23:23:23.992608 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:23.992569    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb" podUID="c4ac6824-07be-4095-82ce-47dfe0ca8098" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:23:30.131580 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:30.131530    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb" podUID="c4ac6824-07be-4095-82ce-47dfe0ca8098" containerName="main" containerID="cri-o://027a9c20f45608bfd7ac4b66df09344e1c7701f9961945a90d1ec52f9055deda" gracePeriod=45
Jul 29 23:23:30.380434 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:30.380408    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"
Jul 29 23:23:30.526271 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:30.526235    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldg9j\" (UniqueName: \"kubernetes.io/projected/c4ac6824-07be-4095-82ce-47dfe0ca8098-kube-api-access-ldg9j\") pod \"c4ac6824-07be-4095-82ce-47dfe0ca8098\" (UID: \"c4ac6824-07be-4095-82ce-47dfe0ca8098\") "
Jul 29 23:23:30.526426 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:30.526296    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ac6824-07be-4095-82ce-47dfe0ca8098-tls-certs\") pod \"c4ac6824-07be-4095-82ce-47dfe0ca8098\" (UID: \"c4ac6824-07be-4095-82ce-47dfe0ca8098\") "
Jul 29 23:23:30.528493 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:30.528442    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4ac6824-07be-4095-82ce-47dfe0ca8098-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c4ac6824-07be-4095-82ce-47dfe0ca8098" (UID: "c4ac6824-07be-4095-82ce-47dfe0ca8098"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:23:30.528493 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:30.528459    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ac6824-07be-4095-82ce-47dfe0ca8098-kube-api-access-ldg9j" (OuterVolumeSpecName: "kube-api-access-ldg9j") pod "c4ac6824-07be-4095-82ce-47dfe0ca8098" (UID: "c4ac6824-07be-4095-82ce-47dfe0ca8098"). InnerVolumeSpecName "kube-api-access-ldg9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:23:30.627744 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:30.627709    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldg9j\" (UniqueName: \"kubernetes.io/projected/c4ac6824-07be-4095-82ce-47dfe0ca8098-kube-api-access-ldg9j\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:23:30.627744 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:30.627736    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ac6824-07be-4095-82ce-47dfe0ca8098-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:23:31.196701 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:31.196666    2569 generic.go:356] "Generic (PLEG): container finished" podID="c4ac6824-07be-4095-82ce-47dfe0ca8098" containerID="027a9c20f45608bfd7ac4b66df09344e1c7701f9961945a90d1ec52f9055deda" exitCode=0
Jul 29 23:23:31.197117 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:31.196712    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"
Jul 29 23:23:31.197117 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:31.196737    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb" event={"ID":"c4ac6824-07be-4095-82ce-47dfe0ca8098","Type":"ContainerDied","Data":"027a9c20f45608bfd7ac4b66df09344e1c7701f9961945a90d1ec52f9055deda"}
Jul 29 23:23:31.197117 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:31.196780    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb" event={"ID":"c4ac6824-07be-4095-82ce-47dfe0ca8098","Type":"ContainerDied","Data":"25fd43a7d3179bb11e85ff228a94b5b04f36ac4a5cd73d2b31dec5375e297e2e"}
Jul 29 23:23:31.197117 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:31.196806    2569 scope.go:117] "RemoveContainer" containerID="027a9c20f45608bfd7ac4b66df09344e1c7701f9961945a90d1ec52f9055deda"
Jul 29 23:23:31.205406 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:31.205388    2569 scope.go:117] "RemoveContainer" containerID="027a9c20f45608bfd7ac4b66df09344e1c7701f9961945a90d1ec52f9055deda"
Jul 29 23:23:31.205682 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:23:31.205664    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027a9c20f45608bfd7ac4b66df09344e1c7701f9961945a90d1ec52f9055deda\": container with ID starting with 027a9c20f45608bfd7ac4b66df09344e1c7701f9961945a90d1ec52f9055deda not found: ID does not exist" containerID="027a9c20f45608bfd7ac4b66df09344e1c7701f9961945a90d1ec52f9055deda"
Jul 29 23:23:31.205749 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:31.205691    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027a9c20f45608bfd7ac4b66df09344e1c7701f9961945a90d1ec52f9055deda"} err="failed to get container status \"027a9c20f45608bfd7ac4b66df09344e1c7701f9961945a90d1ec52f9055deda\": rpc error: code = NotFound desc = could not find container \"027a9c20f45608bfd7ac4b66df09344e1c7701f9961945a90d1ec52f9055deda\": container with ID starting with 027a9c20f45608bfd7ac4b66df09344e1c7701f9961945a90d1ec52f9055deda not found: ID does not exist"
Jul 29 23:23:31.217842 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:31.217821    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"]
Jul 29 23:23:31.221385 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:31.221363    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9mmdb"]
Jul 29 23:23:31.473784 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:23:31.473750    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4ac6824-07be-4095-82ce-47dfe0ca8098" path="/var/lib/kubelet/pods/c4ac6824-07be-4095-82ce-47dfe0ca8098/volumes"
Jul 29 23:24:48.287899 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.287861    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"]
Jul 29 23:24:48.288410 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.288193    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aba005bb-2e30-4c84-9814-2a079c993446" containerName="main"
Jul 29 23:24:48.288410 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.288204    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="aba005bb-2e30-4c84-9814-2a079c993446" containerName="main"
Jul 29 23:24:48.288410 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.288218    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4ac6824-07be-4095-82ce-47dfe0ca8098" containerName="main"
Jul 29 23:24:48.288410 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.288224    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4ac6824-07be-4095-82ce-47dfe0ca8098" containerName="main"
Jul 29 23:24:48.288410 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.288274    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4ac6824-07be-4095-82ce-47dfe0ca8098" containerName="main"
Jul 29 23:24:48.288410 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.288282    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="aba005bb-2e30-4c84-9814-2a079c993446" containerName="main"
Jul 29 23:24:48.291130 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.291108    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.293590 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.293562    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 29 23:24:48.294296 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.294276    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 29 23:24:48.294668 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.294646    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-x9ppl\""
Jul 29 23:24:48.294796 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.294774    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 29 23:24:48.301776 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.301747    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"]
Jul 29 23:24:48.360733 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.360686    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce3e877-dee0-4a11-a72e-e9742f825f2e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.360914 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.360767    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.360914 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.360796    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.360914 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.360816    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.360914 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.360844    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll769\" (UniqueName: \"kubernetes.io/projected/bce3e877-dee0-4a11-a72e-e9742f825f2e-kube-api-access-ll769\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.360914 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.360875    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.461168 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.461131    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.461367 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.461174    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.461367 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.461321    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll769\" (UniqueName: \"kubernetes.io/projected/bce3e877-dee0-4a11-a72e-e9742f825f2e-kube-api-access-ll769\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.461535 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.461395    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.461535 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.461458    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce3e877-dee0-4a11-a72e-e9742f825f2e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.461656 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.461567    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.461656 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.461601    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.461766 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.461746    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.461909 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.461888    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.463505 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.463470    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.463877 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.463858    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce3e877-dee0-4a11-a72e-e9742f825f2e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.469515 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.469473    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll769\" (UniqueName: \"kubernetes.io/projected/bce3e877-dee0-4a11-a72e-e9742f825f2e-kube-api-access-ll769\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.603531 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.603417    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:48.740470 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:48.740442    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"]
Jul 29 23:24:48.742544 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:24:48.742497    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce3e877_dee0_4a11_a72e_e9742f825f2e.slice/crio-d4edb2f55d34995adda93fdf4f58d73d1418b2855e1654129e1f3121383d21ba WatchSource:0}: Error finding container d4edb2f55d34995adda93fdf4f58d73d1418b2855e1654129e1f3121383d21ba: Status 404 returned error can't find the container with id d4edb2f55d34995adda93fdf4f58d73d1418b2855e1654129e1f3121383d21ba
Jul 29 23:24:49.500503 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:49.500449    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q" event={"ID":"bce3e877-dee0-4a11-a72e-e9742f825f2e","Type":"ContainerStarted","Data":"e69e98b2e6a20bdfd071d92e6ceb297d5c47c29ddd8e3dd8dff0d64e55c11877"}
Jul 29 23:24:49.500921 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:49.500526    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q" event={"ID":"bce3e877-dee0-4a11-a72e-e9742f825f2e","Type":"ContainerStarted","Data":"d4edb2f55d34995adda93fdf4f58d73d1418b2855e1654129e1f3121383d21ba"}
Jul 29 23:24:49.519399 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:49.519346    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q" podStartSLOduration=1.519330315 podStartE2EDuration="1.519330315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:24:49.518273624 +0000 UTC m=+2296.523471048" watchObservedRunningTime="2026-07-29 23:24:49.519330315 +0000 UTC m=+2296.524527738"
Jul 29 23:24:49.604454 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:49.604415    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:58.604623 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:58.604589    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:58.617013 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:58.616986    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:24:59.550495 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:24:59.550452    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:25:22.211211 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.210809    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"]
Jul 29 23:25:22.221299 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:25:22.221266    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:25:22.221299 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:25:22.221299 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:25:22.221299 ip-10-0-135-98 kubenswrapper[2569]:  > containerID="e69e98b2e6a20bdfd071d92e6ceb297d5c47c29ddd8e3dd8dff0d64e55c11877" cmd=["/bin/sleep","15"]
Jul 29 23:25:22.221575 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.221316    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q" message="" err=<
Jul 29 23:25:22.221575 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:25:22.221575 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:25:22.221575 ip-10-0-135-98 kubenswrapper[2569]:  >
Jul 29 23:25:22.221575 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:25:22.221330    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:25:22.221575 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:25:22.221575 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 23:25:22.221575 ip-10-0-135-98 kubenswrapper[2569]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q" podUID="bce3e877-dee0-4a11-a72e-e9742f825f2e" containerName="main" containerID="cri-o://e69e98b2e6a20bdfd071d92e6ceb297d5c47c29ddd8e3dd8dff0d64e55c11877"
Jul 29 23:25:22.221575 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.221368    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q" podUID="bce3e877-dee0-4a11-a72e-e9742f825f2e" containerName="main" containerID="cri-o://e69e98b2e6a20bdfd071d92e6ceb297d5c47c29ddd8e3dd8dff0d64e55c11877" gracePeriod=60
Jul 29 23:25:22.572274 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.572251    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:25:22.636727 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.636690    2569 generic.go:356] "Generic (PLEG): container finished" podID="bce3e877-dee0-4a11-a72e-e9742f825f2e" containerID="e69e98b2e6a20bdfd071d92e6ceb297d5c47c29ddd8e3dd8dff0d64e55c11877" exitCode=0
Jul 29 23:25:22.636876 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.636731    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"
Jul 29 23:25:22.636876 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.636765    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q" event={"ID":"bce3e877-dee0-4a11-a72e-e9742f825f2e","Type":"ContainerDied","Data":"e69e98b2e6a20bdfd071d92e6ceb297d5c47c29ddd8e3dd8dff0d64e55c11877"}
Jul 29 23:25:22.636876 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.636813    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q" event={"ID":"bce3e877-dee0-4a11-a72e-e9742f825f2e","Type":"ContainerDied","Data":"d4edb2f55d34995adda93fdf4f58d73d1418b2855e1654129e1f3121383d21ba"}
Jul 29 23:25:22.636876 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.636837    2569 scope.go:117] "RemoveContainer" containerID="e69e98b2e6a20bdfd071d92e6ceb297d5c47c29ddd8e3dd8dff0d64e55c11877"
Jul 29 23:25:22.645375 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.645352    2569 scope.go:117] "RemoveContainer" containerID="e69e98b2e6a20bdfd071d92e6ceb297d5c47c29ddd8e3dd8dff0d64e55c11877"
Jul 29 23:25:22.645711 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:25:22.645690    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69e98b2e6a20bdfd071d92e6ceb297d5c47c29ddd8e3dd8dff0d64e55c11877\": container with ID starting with e69e98b2e6a20bdfd071d92e6ceb297d5c47c29ddd8e3dd8dff0d64e55c11877 not found: ID does not exist" containerID="e69e98b2e6a20bdfd071d92e6ceb297d5c47c29ddd8e3dd8dff0d64e55c11877"
Jul 29 23:25:22.645762 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.645723    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69e98b2e6a20bdfd071d92e6ceb297d5c47c29ddd8e3dd8dff0d64e55c11877"} err="failed to get container status \"e69e98b2e6a20bdfd071d92e6ceb297d5c47c29ddd8e3dd8dff0d64e55c11877\": rpc error: code = NotFound desc = could not find container \"e69e98b2e6a20bdfd071d92e6ceb297d5c47c29ddd8e3dd8dff0d64e55c11877\": container with ID starting with e69e98b2e6a20bdfd071d92e6ceb297d5c47c29ddd8e3dd8dff0d64e55c11877 not found: ID does not exist"
Jul 29 23:25:22.670586 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.670555    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-home\") pod \"bce3e877-dee0-4a11-a72e-e9742f825f2e\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") "
Jul 29 23:25:22.670720 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.670613    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-dshm\") pod \"bce3e877-dee0-4a11-a72e-e9742f825f2e\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") "
Jul 29 23:25:22.670720 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.670641    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-tmp-dir\") pod \"bce3e877-dee0-4a11-a72e-e9742f825f2e\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") "
Jul 29 23:25:22.670720 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.670663    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce3e877-dee0-4a11-a72e-e9742f825f2e-tls-certs\") pod \"bce3e877-dee0-4a11-a72e-e9742f825f2e\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") "
Jul 29 23:25:22.670720 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.670682    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll769\" (UniqueName: \"kubernetes.io/projected/bce3e877-dee0-4a11-a72e-e9742f825f2e-kube-api-access-ll769\") pod \"bce3e877-dee0-4a11-a72e-e9742f825f2e\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") "
Jul 29 23:25:22.670892 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.670770    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-model-cache\") pod \"bce3e877-dee0-4a11-a72e-e9742f825f2e\" (UID: \"bce3e877-dee0-4a11-a72e-e9742f825f2e\") "
Jul 29 23:25:22.670892 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.670867    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-home" (OuterVolumeSpecName: "home") pod "bce3e877-dee0-4a11-a72e-e9742f825f2e" (UID: "bce3e877-dee0-4a11-a72e-e9742f825f2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:25:22.670999 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.670923    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bce3e877-dee0-4a11-a72e-e9742f825f2e" (UID: "bce3e877-dee0-4a11-a72e-e9742f825f2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:25:22.671065 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.671017    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:25:22.671065 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.671031    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:25:22.671169 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.671082    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-model-cache" (OuterVolumeSpecName: "model-cache") pod "bce3e877-dee0-4a11-a72e-e9742f825f2e" (UID: "bce3e877-dee0-4a11-a72e-e9742f825f2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:25:22.672971 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.672940    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce3e877-dee0-4a11-a72e-e9742f825f2e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bce3e877-dee0-4a11-a72e-e9742f825f2e" (UID: "bce3e877-dee0-4a11-a72e-e9742f825f2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:25:22.673091 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.672976    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce3e877-dee0-4a11-a72e-e9742f825f2e-kube-api-access-ll769" (OuterVolumeSpecName: "kube-api-access-ll769") pod "bce3e877-dee0-4a11-a72e-e9742f825f2e" (UID: "bce3e877-dee0-4a11-a72e-e9742f825f2e"). InnerVolumeSpecName "kube-api-access-ll769". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:25:22.673091 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.673028    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-dshm" (OuterVolumeSpecName: "dshm") pod "bce3e877-dee0-4a11-a72e-e9742f825f2e" (UID: "bce3e877-dee0-4a11-a72e-e9742f825f2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:25:22.772052 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.771966    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:25:22.772052 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.771996    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce3e877-dee0-4a11-a72e-e9742f825f2e-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:25:22.772052 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.772005    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce3e877-dee0-4a11-a72e-e9742f825f2e-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:25:22.772052 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.772014    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll769\" (UniqueName: \"kubernetes.io/projected/bce3e877-dee0-4a11-a72e-e9742f825f2e-kube-api-access-ll769\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:25:22.958296 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.958265    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"]
Jul 29 23:25:22.962363 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:22.962335    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q"]
Jul 29 23:25:23.474044 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:23.474009    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bce3e877-dee0-4a11-a72e-e9742f825f2e" path="/var/lib/kubelet/pods/bce3e877-dee0-4a11-a72e-e9742f825f2e/volumes"
Jul 29 23:25:23.540344 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:25:23.540295    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-pp59q" podUID="bce3e877-dee0-4a11-a72e-e9742f825f2e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded"
Jul 29 23:26:09.328841 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.328798    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"]
Jul 29 23:26:09.329465 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.329251    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce3e877-dee0-4a11-a72e-e9742f825f2e" containerName="main"
Jul 29 23:26:09.329465 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.329271    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce3e877-dee0-4a11-a72e-e9742f825f2e" containerName="main"
Jul 29 23:26:09.329465 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.329353    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce3e877-dee0-4a11-a72e-e9742f825f2e" containerName="main"
Jul 29 23:26:09.332531 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.332507    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"
Jul 29 23:26:09.335315 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.335273    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-w7qz4\""
Jul 29 23:26:09.335315 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.335290    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 29 23:26:09.335315 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.335300    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 29 23:26:09.335623 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.335392    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 29 23:26:09.335623 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.335427    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-7r2pc\""
Jul 29 23:26:09.340273 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.340190    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"]
Jul 29 23:26:09.451202 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.451158    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv7zn\" (UniqueName: \"kubernetes.io/projected/641aba9d-a6c6-4a86-809b-df5a6f8c714b-kube-api-access-fv7zn\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq\" (UID: \"641aba9d-a6c6-4a86-809b-df5a6f8c714b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"
Jul 29 23:26:09.451394 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.451219    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641aba9d-a6c6-4a86-809b-df5a6f8c714b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq\" (UID: \"641aba9d-a6c6-4a86-809b-df5a6f8c714b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"
Jul 29 23:26:09.552529 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.552467    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv7zn\" (UniqueName: \"kubernetes.io/projected/641aba9d-a6c6-4a86-809b-df5a6f8c714b-kube-api-access-fv7zn\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq\" (UID: \"641aba9d-a6c6-4a86-809b-df5a6f8c714b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"
Jul 29 23:26:09.552703 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.552546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641aba9d-a6c6-4a86-809b-df5a6f8c714b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq\" (UID: \"641aba9d-a6c6-4a86-809b-df5a6f8c714b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"
Jul 29 23:26:09.555211 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.555191    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641aba9d-a6c6-4a86-809b-df5a6f8c714b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq\" (UID: \"641aba9d-a6c6-4a86-809b-df5a6f8c714b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"
Jul 29 23:26:09.560122 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.560094    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv7zn\" (UniqueName: \"kubernetes.io/projected/641aba9d-a6c6-4a86-809b-df5a6f8c714b-kube-api-access-fv7zn\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq\" (UID: \"641aba9d-a6c6-4a86-809b-df5a6f8c714b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"
Jul 29 23:26:09.644212 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.644126    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"
Jul 29 23:26:09.773570 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.773543    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"]
Jul 29 23:26:09.776035 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:26:09.775999    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641aba9d_a6c6_4a86_809b_df5a6f8c714b.slice/crio-0fddddc8fea1dd604aed7f230f2ae86f27d7736b52a6b8dd4898c6dabfc4ef5f WatchSource:0}: Error finding container 0fddddc8fea1dd604aed7f230f2ae86f27d7736b52a6b8dd4898c6dabfc4ef5f: Status 404 returned error can't find the container with id 0fddddc8fea1dd604aed7f230f2ae86f27d7736b52a6b8dd4898c6dabfc4ef5f
Jul 29 23:26:09.820307 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:09.820278    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq" event={"ID":"641aba9d-a6c6-4a86-809b-df5a6f8c714b","Type":"ContainerStarted","Data":"0fddddc8fea1dd604aed7f230f2ae86f27d7736b52a6b8dd4898c6dabfc4ef5f"}
Jul 29 23:26:10.825843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:10.825808    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq" event={"ID":"641aba9d-a6c6-4a86-809b-df5a6f8c714b","Type":"ContainerStarted","Data":"2b14ba43071b8ec57e056c9dca5c9949914d4e257e1e1faae82a7e035bd0ad5b"}
Jul 29 23:26:10.826264 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:10.825947    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"
Jul 29 23:26:10.841873 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:10.841825    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq" podStartSLOduration=1.841810245 podStartE2EDuration="1.841810245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:26:10.841509827 +0000 UTC m=+2377.846707247" watchObservedRunningTime="2026-07-29 23:26:10.841810245 +0000 UTC m=+2377.847007667"
Jul 29 23:26:33.598910 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:33.598869    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:26:33.606133 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:33.606109    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:26:41.831909 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:26:41.831877    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"
Jul 29 23:28:04.357235 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:04.357198    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"]
Jul 29 23:28:11.831989 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:11.831949    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq" podUID="641aba9d-a6c6-4a86-809b-df5a6f8c714b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:28:19.377132 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:19.377066    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq" podUID="641aba9d-a6c6-4a86-809b-df5a6f8c714b" containerName="main" containerID="cri-o://2b14ba43071b8ec57e056c9dca5c9949914d4e257e1e1faae82a7e035bd0ad5b" gracePeriod=45
Jul 29 23:28:19.621175 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:19.621144    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"
Jul 29 23:28:19.695205 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:19.695110    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641aba9d-a6c6-4a86-809b-df5a6f8c714b-tls-certs\") pod \"641aba9d-a6c6-4a86-809b-df5a6f8c714b\" (UID: \"641aba9d-a6c6-4a86-809b-df5a6f8c714b\") "
Jul 29 23:28:19.695205 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:19.695155    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv7zn\" (UniqueName: \"kubernetes.io/projected/641aba9d-a6c6-4a86-809b-df5a6f8c714b-kube-api-access-fv7zn\") pod \"641aba9d-a6c6-4a86-809b-df5a6f8c714b\" (UID: \"641aba9d-a6c6-4a86-809b-df5a6f8c714b\") "
Jul 29 23:28:19.697355 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:19.697326    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641aba9d-a6c6-4a86-809b-df5a6f8c714b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "641aba9d-a6c6-4a86-809b-df5a6f8c714b" (UID: "641aba9d-a6c6-4a86-809b-df5a6f8c714b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:28:19.697475 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:19.697400    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641aba9d-a6c6-4a86-809b-df5a6f8c714b-kube-api-access-fv7zn" (OuterVolumeSpecName: "kube-api-access-fv7zn") pod "641aba9d-a6c6-4a86-809b-df5a6f8c714b" (UID: "641aba9d-a6c6-4a86-809b-df5a6f8c714b"). InnerVolumeSpecName "kube-api-access-fv7zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:28:19.796394 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:19.796355    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641aba9d-a6c6-4a86-809b-df5a6f8c714b-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:28:19.796585 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:19.796404    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv7zn\" (UniqueName: \"kubernetes.io/projected/641aba9d-a6c6-4a86-809b-df5a6f8c714b-kube-api-access-fv7zn\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:28:20.342295 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:20.342258    2569 generic.go:356] "Generic (PLEG): container finished" podID="641aba9d-a6c6-4a86-809b-df5a6f8c714b" containerID="2b14ba43071b8ec57e056c9dca5c9949914d4e257e1e1faae82a7e035bd0ad5b" exitCode=0
Jul 29 23:28:20.342538 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:20.342308    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"
Jul 29 23:28:20.342538 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:20.342311    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq" event={"ID":"641aba9d-a6c6-4a86-809b-df5a6f8c714b","Type":"ContainerDied","Data":"2b14ba43071b8ec57e056c9dca5c9949914d4e257e1e1faae82a7e035bd0ad5b"}
Jul 29 23:28:20.342538 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:20.342434    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq" event={"ID":"641aba9d-a6c6-4a86-809b-df5a6f8c714b","Type":"ContainerDied","Data":"0fddddc8fea1dd604aed7f230f2ae86f27d7736b52a6b8dd4898c6dabfc4ef5f"}
Jul 29 23:28:20.342538 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:20.342463    2569 scope.go:117] "RemoveContainer" containerID="2b14ba43071b8ec57e056c9dca5c9949914d4e257e1e1faae82a7e035bd0ad5b"
Jul 29 23:28:20.350886 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:20.350868    2569 scope.go:117] "RemoveContainer" containerID="2b14ba43071b8ec57e056c9dca5c9949914d4e257e1e1faae82a7e035bd0ad5b"
Jul 29 23:28:20.351144 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:28:20.351125    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b14ba43071b8ec57e056c9dca5c9949914d4e257e1e1faae82a7e035bd0ad5b\": container with ID starting with 2b14ba43071b8ec57e056c9dca5c9949914d4e257e1e1faae82a7e035bd0ad5b not found: ID does not exist" containerID="2b14ba43071b8ec57e056c9dca5c9949914d4e257e1e1faae82a7e035bd0ad5b"
Jul 29 23:28:20.351198 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:20.351152    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b14ba43071b8ec57e056c9dca5c9949914d4e257e1e1faae82a7e035bd0ad5b"} err="failed to get container status \"2b14ba43071b8ec57e056c9dca5c9949914d4e257e1e1faae82a7e035bd0ad5b\": rpc error: code = NotFound desc = could not find container \"2b14ba43071b8ec57e056c9dca5c9949914d4e257e1e1faae82a7e035bd0ad5b\": container with ID starting with 2b14ba43071b8ec57e056c9dca5c9949914d4e257e1e1faae82a7e035bd0ad5b not found: ID does not exist"
Jul 29 23:28:20.362088 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:20.362064    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"]
Jul 29 23:28:20.368022 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:20.367989    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schescdwq"]
Jul 29 23:28:21.473726 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:21.473693    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="641aba9d-a6c6-4a86-809b-df5a6f8c714b" path="/var/lib/kubelet/pods/641aba9d-a6c6-4a86-809b-df5a6f8c714b/volumes"
Jul 29 23:28:28.437632 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.437527    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"]
Jul 29 23:28:28.438109 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.437973    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="641aba9d-a6c6-4a86-809b-df5a6f8c714b" containerName="main"
Jul 29 23:28:28.438109 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.437992    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="641aba9d-a6c6-4a86-809b-df5a6f8c714b" containerName="main"
Jul 29 23:28:28.438109 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.438095    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="641aba9d-a6c6-4a86-809b-df5a6f8c714b" containerName="main"
Jul 29 23:28:28.442587 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.442564    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"
Jul 29 23:28:28.445754 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.445423    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-l66kp\""
Jul 29 23:28:28.445754 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.445476    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 29 23:28:28.445754 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.445534    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-vf9bf\""
Jul 29 23:28:28.445754 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.445431    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 29 23:28:28.445754 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.445644    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 29 23:28:28.448338 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.448316    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"]
Jul 29 23:28:28.568327 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.568285    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218cc495-00a4-4611-aa92-0718c52deae9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk\" (UID: \"218cc495-00a4-4611-aa92-0718c52deae9\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"
Jul 29 23:28:28.568559 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.568425    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5fsf\" (UniqueName: \"kubernetes.io/projected/218cc495-00a4-4611-aa92-0718c52deae9-kube-api-access-j5fsf\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk\" (UID: \"218cc495-00a4-4611-aa92-0718c52deae9\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"
Jul 29 23:28:28.669946 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.669901    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5fsf\" (UniqueName: \"kubernetes.io/projected/218cc495-00a4-4611-aa92-0718c52deae9-kube-api-access-j5fsf\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk\" (UID: \"218cc495-00a4-4611-aa92-0718c52deae9\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"
Jul 29 23:28:28.670156 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.669957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218cc495-00a4-4611-aa92-0718c52deae9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk\" (UID: \"218cc495-00a4-4611-aa92-0718c52deae9\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"
Jul 29 23:28:28.672455 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.672428    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218cc495-00a4-4611-aa92-0718c52deae9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk\" (UID: \"218cc495-00a4-4611-aa92-0718c52deae9\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"
Jul 29 23:28:28.678004 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.677980    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5fsf\" (UniqueName: \"kubernetes.io/projected/218cc495-00a4-4611-aa92-0718c52deae9-kube-api-access-j5fsf\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk\" (UID: \"218cc495-00a4-4611-aa92-0718c52deae9\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"
Jul 29 23:28:28.755110 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.755069    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"
Jul 29 23:28:28.889010 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:28.888983    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"]
Jul 29 23:28:28.891578 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:28:28.891546    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218cc495_00a4_4611_aa92_0718c52deae9.slice/crio-a9b793b1f8a01bbbd7287062ba09cda396617b3b3da6261721bff20bfa18d054 WatchSource:0}: Error finding container a9b793b1f8a01bbbd7287062ba09cda396617b3b3da6261721bff20bfa18d054: Status 404 returned error can't find the container with id a9b793b1f8a01bbbd7287062ba09cda396617b3b3da6261721bff20bfa18d054
Jul 29 23:28:29.378895 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:29.378853    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk" event={"ID":"218cc495-00a4-4611-aa92-0718c52deae9","Type":"ContainerStarted","Data":"70d9dbe1611e50bd7f11dc2482af14e43b36b46d74070d0cc42221ed28228617"}
Jul 29 23:28:29.379117 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:29.378948    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"
Jul 29 23:28:29.379117 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:29.378965    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk" event={"ID":"218cc495-00a4-4611-aa92-0718c52deae9","Type":"ContainerStarted","Data":"a9b793b1f8a01bbbd7287062ba09cda396617b3b3da6261721bff20bfa18d054"}
Jul 29 23:28:29.395231 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:28:29.395144    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk" podStartSLOduration=1.3951247709999999 podStartE2EDuration="1.395124771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:28:29.394549985 +0000 UTC m=+2516.399747408" watchObservedRunningTime="2026-07-29 23:28:29.395124771 +0000 UTC m=+2516.400322195"
Jul 29 23:29:00.385973 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:29:00.385944    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"
Jul 29 23:30:00.162093 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:00.161989    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"]
Jul 29 23:30:00.385121 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:00.385082    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk" podUID="218cc495-00a4-4611-aa92-0718c52deae9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:30:10.385639 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:10.385597    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk" podUID="218cc495-00a4-4611-aa92-0718c52deae9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:30:15.181499 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.181411    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk" podUID="218cc495-00a4-4611-aa92-0718c52deae9" containerName="main" containerID="cri-o://70d9dbe1611e50bd7f11dc2482af14e43b36b46d74070d0cc42221ed28228617" gracePeriod=45
Jul 29 23:30:15.431935 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.431871    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"
Jul 29 23:30:15.472451 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.472419    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5fsf\" (UniqueName: \"kubernetes.io/projected/218cc495-00a4-4611-aa92-0718c52deae9-kube-api-access-j5fsf\") pod \"218cc495-00a4-4611-aa92-0718c52deae9\" (UID: \"218cc495-00a4-4611-aa92-0718c52deae9\") "
Jul 29 23:30:15.472630 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.472516    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218cc495-00a4-4611-aa92-0718c52deae9-tls-certs\") pod \"218cc495-00a4-4611-aa92-0718c52deae9\" (UID: \"218cc495-00a4-4611-aa92-0718c52deae9\") "
Jul 29 23:30:15.474655 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.474628    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218cc495-00a4-4611-aa92-0718c52deae9-kube-api-access-j5fsf" (OuterVolumeSpecName: "kube-api-access-j5fsf") pod "218cc495-00a4-4611-aa92-0718c52deae9" (UID: "218cc495-00a4-4611-aa92-0718c52deae9"). InnerVolumeSpecName "kube-api-access-j5fsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:30:15.474655 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.474636    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218cc495-00a4-4611-aa92-0718c52deae9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "218cc495-00a4-4611-aa92-0718c52deae9" (UID: "218cc495-00a4-4611-aa92-0718c52deae9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:30:15.573755 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.573717    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5fsf\" (UniqueName: \"kubernetes.io/projected/218cc495-00a4-4611-aa92-0718c52deae9-kube-api-access-j5fsf\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:30:15.573755 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.573753    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218cc495-00a4-4611-aa92-0718c52deae9-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:30:15.801737 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.801693    2569 generic.go:356] "Generic (PLEG): container finished" podID="218cc495-00a4-4611-aa92-0718c52deae9" containerID="70d9dbe1611e50bd7f11dc2482af14e43b36b46d74070d0cc42221ed28228617" exitCode=0
Jul 29 23:30:15.801737 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.801736    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk" event={"ID":"218cc495-00a4-4611-aa92-0718c52deae9","Type":"ContainerDied","Data":"70d9dbe1611e50bd7f11dc2482af14e43b36b46d74070d0cc42221ed28228617"}
Jul 29 23:30:15.802004 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.801742    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"
Jul 29 23:30:15.802004 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.801777    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk" event={"ID":"218cc495-00a4-4611-aa92-0718c52deae9","Type":"ContainerDied","Data":"a9b793b1f8a01bbbd7287062ba09cda396617b3b3da6261721bff20bfa18d054"}
Jul 29 23:30:15.802004 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.801801    2569 scope.go:117] "RemoveContainer" containerID="70d9dbe1611e50bd7f11dc2482af14e43b36b46d74070d0cc42221ed28228617"
Jul 29 23:30:15.811273 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.811124    2569 scope.go:117] "RemoveContainer" containerID="70d9dbe1611e50bd7f11dc2482af14e43b36b46d74070d0cc42221ed28228617"
Jul 29 23:30:15.811458 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:30:15.811430    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d9dbe1611e50bd7f11dc2482af14e43b36b46d74070d0cc42221ed28228617\": container with ID starting with 70d9dbe1611e50bd7f11dc2482af14e43b36b46d74070d0cc42221ed28228617 not found: ID does not exist" containerID="70d9dbe1611e50bd7f11dc2482af14e43b36b46d74070d0cc42221ed28228617"
Jul 29 23:30:15.811579 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.811467    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d9dbe1611e50bd7f11dc2482af14e43b36b46d74070d0cc42221ed28228617"} err="failed to get container status \"70d9dbe1611e50bd7f11dc2482af14e43b36b46d74070d0cc42221ed28228617\": rpc error: code = NotFound desc = could not find container \"70d9dbe1611e50bd7f11dc2482af14e43b36b46d74070d0cc42221ed28228617\": container with ID starting with 70d9dbe1611e50bd7f11dc2482af14e43b36b46d74070d0cc42221ed28228617 not found: ID does not exist"
Jul 29 23:30:15.823759 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.823730    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"]
Jul 29 23:30:15.827078 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:15.827053    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schehstvk"]
Jul 29 23:30:17.473985 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:30:17.473949    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="218cc495-00a4-4611-aa92-0718c52deae9" path="/var/lib/kubelet/pods/218cc495-00a4-4611-aa92-0718c52deae9/volumes"
Jul 29 23:31:33.627667 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:31:33.623311    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:31:33.637918 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:31:33.637894    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:33:59.756932 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.756892    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv"]
Jul 29 23:33:59.757455 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.757392    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="218cc495-00a4-4611-aa92-0718c52deae9" containerName="main"
Jul 29 23:33:59.757455 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.757410    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="218cc495-00a4-4611-aa92-0718c52deae9" containerName="main"
Jul 29 23:33:59.757565 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.757530    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="218cc495-00a4-4611-aa92-0718c52deae9" containerName="main"
Jul 29 23:33:59.760722 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.760697    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv"
Jul 29 23:33:59.763499 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.763440    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-sdqb6\""
Jul 29 23:33:59.763499 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.763440    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 29 23:33:59.763651 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.763448    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 29 23:33:59.763651 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.763449    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-w25b2\""
Jul 29 23:33:59.763651 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.763463    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 29 23:33:59.768566 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.768545    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv"]
Jul 29 23:33:59.808367 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.808328    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv\" (UID: \"7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv"
Jul 29 23:33:59.808583 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.808464    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr9zk\" (UniqueName: \"kubernetes.io/projected/7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b-kube-api-access-lr9zk\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv\" (UID: \"7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv"
Jul 29 23:33:59.909477 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.909440    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv\" (UID: \"7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv"
Jul 29 23:33:59.909694 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.909560    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr9zk\" (UniqueName: \"kubernetes.io/projected/7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b-kube-api-access-lr9zk\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv\" (UID: \"7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv"
Jul 29 23:33:59.911933 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.911911    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv\" (UID: \"7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv"
Jul 29 23:33:59.916583 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:33:59.916556    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr9zk\" (UniqueName: \"kubernetes.io/projected/7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b-kube-api-access-lr9zk\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv\" (UID: \"7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv"
Jul 29 23:34:00.072226 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:34:00.072127    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv"
Jul 29 23:34:00.205452 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:34:00.205307    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv"]
Jul 29 23:34:00.208469 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:34:00.208435    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0ebd49_e60f_4463_9b0d_2b74cbc7e69b.slice/crio-1c916367bcd97c8d1d2a90cb534a8d81cb03401363afdf27d0589529bc664508 WatchSource:0}: Error finding container 1c916367bcd97c8d1d2a90cb534a8d81cb03401363afdf27d0589529bc664508: Status 404 returned error can't find the container with id 1c916367bcd97c8d1d2a90cb534a8d81cb03401363afdf27d0589529bc664508
Jul 29 23:34:00.684101 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:34:00.684066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv" event={"ID":"7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b","Type":"ContainerStarted","Data":"49206a0fd920627fddb6f39e965dcbdc49b31ad51fedbc12de77766b3667b4ee"}
Jul 29 23:34:00.684280 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:34:00.684118    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv" event={"ID":"7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b","Type":"ContainerStarted","Data":"1c916367bcd97c8d1d2a90cb534a8d81cb03401363afdf27d0589529bc664508"}
Jul 29 23:34:00.684280 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:34:00.684195    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv"
Jul 29 23:34:00.699025 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:34:00.698967    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv" podStartSLOduration=1.698950886 podStartE2EDuration="1.698950886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:34:00.697647291 +0000 UTC m=+2847.702844717" watchObservedRunningTime="2026-07-29 23:34:00.698950886 +0000 UTC m=+2847.704148308"
Jul 29 23:34:31.689610 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:34:31.689527    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv"
Jul 29 23:36:33.654535 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:36:33.654501    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:36:33.662953 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:36:33.662930    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:41:33.679153 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:41:33.679119    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:41:33.687973 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:41:33.687947    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:46:33.703412 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:46:33.703300    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:46:33.713006 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:46:33.712984    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:48:44.817115 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:44.817087    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv_7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b/main/0.log"
Jul 29 23:48:49.357315 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:49.357269    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"]
Jul 29 23:48:49.360670 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:49.360631    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"
Jul 29 23:48:49.362670 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:49.362637    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 29 23:48:49.362670 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:49.362639    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 29 23:48:49.363305 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:49.363282    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-vxspr\""
Jul 29 23:48:49.372184 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:49.372157    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"]
Jul 29 23:48:49.420057 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:49.420022    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tw2r\" (UniqueName: \"kubernetes.io/projected/30e0f470-dbcb-43b1-a947-652831e32f74-kube-api-access-7tw2r\") pod \"e2e-pvc-model-download-8m5qp\" (UID: \"30e0f470-dbcb-43b1-a947-652831e32f74\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"
Jul 29 23:48:49.520942 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:49.520911    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tw2r\" (UniqueName: \"kubernetes.io/projected/30e0f470-dbcb-43b1-a947-652831e32f74-kube-api-access-7tw2r\") pod \"e2e-pvc-model-download-8m5qp\" (UID: \"30e0f470-dbcb-43b1-a947-652831e32f74\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"
Jul 29 23:48:49.528099 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:49.528072    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tw2r\" (UniqueName: \"kubernetes.io/projected/30e0f470-dbcb-43b1-a947-652831e32f74-kube-api-access-7tw2r\") pod \"e2e-pvc-model-download-8m5qp\" (UID: \"30e0f470-dbcb-43b1-a947-652831e32f74\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"
Jul 29 23:48:51.743584 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:51.743543    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"e2e-pvc-model-download-8m5qp\" (UID: \"30e0f470-dbcb-43b1-a947-652831e32f74\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"
Jul 29 23:48:51.748219 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:51.748190    2569 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"e2e-pvc-model-download-8m5qp\" (UID: \"30e0f470-dbcb-43b1-a947-652831e32f74\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"
Jul 29 23:48:51.844705 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:51.844655    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"e2e-pvc-model-download-8m5qp\" (UID: \"30e0f470-dbcb-43b1-a947-652831e32f74\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"
Jul 29 23:48:51.844927 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:51.844828    2569 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"e2e-pvc-model-download-8m5qp\" (UID: \"30e0f470-dbcb-43b1-a947-652831e32f74\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"
Jul 29 23:48:51.847435 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:51.847410    2569 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"e2e-pvc-model-download-8m5qp\" (UID: \"30e0f470-dbcb-43b1-a947-652831e32f74\") DevicePath \"csi-01e7d1e59909e242017b8d39295dd7db4f28fee4332177fd46e940fea6dc042c\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"
Jul 29 23:48:52.003809 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:52.003720    2569 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"e2e-pvc-model-download-8m5qp\" (UID: \"30e0f470-dbcb-43b1-a947-652831e32f74\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4ece3d96f092e7290ccd867b9e9ab8f9752db83e65e9e43e089f057ec23de585/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"
Jul 29 23:48:52.017432 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:52.017404    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"e2e-pvc-model-download-8m5qp\" (UID: \"30e0f470-dbcb-43b1-a947-652831e32f74\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"
Jul 29 23:48:52.070364 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:52.070327    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"
Jul 29 23:48:52.202273 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:52.202157    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"]
Jul 29 23:48:52.205112 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:48:52.205085    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30e0f470_dbcb_43b1_a947_652831e32f74.slice/crio-43ce3802e9121fb41fd28a700a26b07b39b44d80b225beb668edd4ec2aee908d WatchSource:0}: Error finding container 43ce3802e9121fb41fd28a700a26b07b39b44d80b225beb668edd4ec2aee908d: Status 404 returned error can't find the container with id 43ce3802e9121fb41fd28a700a26b07b39b44d80b225beb668edd4ec2aee908d
Jul 29 23:48:53.209893 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:53.209779    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp" event={"ID":"30e0f470-dbcb-43b1-a947-652831e32f74","Type":"ContainerStarted","Data":"7a3d8eda4e1e73efd255255224695dafddb394424148b922b43d11a336e62113"}
Jul 29 23:48:53.209893 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:53.209846    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp" event={"ID":"30e0f470-dbcb-43b1-a947-652831e32f74","Type":"ContainerStarted","Data":"43ce3802e9121fb41fd28a700a26b07b39b44d80b225beb668edd4ec2aee908d"}
Jul 29 23:48:53.227326 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:53.227106    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp" podStartSLOduration=7.227084599 podStartE2EDuration="7.227084599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 23:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:48:53.224623245 +0000 UTC m=+3740.229820671" watchObservedRunningTime="2026-07-29 23:48:53.227084599 +0000 UTC m=+3740.232282024"
Jul 29 23:48:59.237439 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:59.237402    2569 generic.go:356] "Generic (PLEG): container finished" podID="30e0f470-dbcb-43b1-a947-652831e32f74" containerID="7a3d8eda4e1e73efd255255224695dafddb394424148b922b43d11a336e62113" exitCode=0
Jul 29 23:48:59.237875 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:48:59.237493    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp" event={"ID":"30e0f470-dbcb-43b1-a947-652831e32f74","Type":"ContainerDied","Data":"7a3d8eda4e1e73efd255255224695dafddb394424148b922b43d11a336e62113"}
Jul 29 23:49:00.389921 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:00.389891    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"
Jul 29 23:49:00.524184 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:00.524099    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"30e0f470-dbcb-43b1-a947-652831e32f74\" (UID: \"30e0f470-dbcb-43b1-a947-652831e32f74\") "
Jul 29 23:49:00.524184 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:00.524168    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tw2r\" (UniqueName: \"kubernetes.io/projected/30e0f470-dbcb-43b1-a947-652831e32f74-kube-api-access-7tw2r\") pod \"30e0f470-dbcb-43b1-a947-652831e32f74\" (UID: \"30e0f470-dbcb-43b1-a947-652831e32f74\") "
Jul 29 23:49:00.526399 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:00.526368    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e0f470-dbcb-43b1-a947-652831e32f74-kube-api-access-7tw2r" (OuterVolumeSpecName: "kube-api-access-7tw2r") pod "30e0f470-dbcb-43b1-a947-652831e32f74" (UID: "30e0f470-dbcb-43b1-a947-652831e32f74"). InnerVolumeSpecName "kube-api-access-7tw2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:49:00.528376 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:00.528342    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222" (OuterVolumeSpecName: "model-storage") pod "30e0f470-dbcb-43b1-a947-652831e32f74" (UID: "30e0f470-dbcb-43b1-a947-652831e32f74"). InnerVolumeSpecName "pvc-380fae1b-9e30-418e-9470-637495923811". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 29 23:49:00.625274 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:00.625244    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tw2r\" (UniqueName: \"kubernetes.io/projected/30e0f470-dbcb-43b1-a947-652831e32f74-kube-api-access-7tw2r\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:49:00.625429 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:00.625292    2569 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") on node \"ip-10-0-135-98.ec2.internal\" "
Jul 29 23:49:00.738724 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:00.738696    2569 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-380fae1b-9e30-418e-9470-637495923811" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222") on node "ip-10-0-135-98.ec2.internal"
Jul 29 23:49:00.827055 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:00.826971    2569 reconciler_common.go:299] "Volume detached for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"csi-01e7d1e59909e242017b8d39295dd7db4f28fee4332177fd46e940fea6dc042c\""
Jul 29 23:49:01.245957 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:01.245920    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp" event={"ID":"30e0f470-dbcb-43b1-a947-652831e32f74","Type":"ContainerDied","Data":"43ce3802e9121fb41fd28a700a26b07b39b44d80b225beb668edd4ec2aee908d"}
Jul 29 23:49:01.245957 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:01.245954    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"
Jul 29 23:49:01.245957 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:01.245963    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43ce3802e9121fb41fd28a700a26b07b39b44d80b225beb668edd4ec2aee908d"
Jul 29 23:49:18.436311 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.436274    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 29 23:49:18.436835 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.436604    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e0f470-dbcb-43b1-a947-652831e32f74" containerName="storage-initializer"
Jul 29 23:49:18.436835 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.436616    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e0f470-dbcb-43b1-a947-652831e32f74" containerName="storage-initializer"
Jul 29 23:49:18.436835 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.436670    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="30e0f470-dbcb-43b1-a947-652831e32f74" containerName="storage-initializer"
Jul 29 23:49:18.439856 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.439831    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.442368 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.442341    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 29 23:49:18.442535 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.442341    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-vxspr\""
Jul 29 23:49:18.442535 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.442385    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 29 23:49:18.442535 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.442397    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 29 23:49:18.442535 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.442342    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-fh7hr\""
Jul 29 23:49:18.450551 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.450526    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 29 23:49:18.593668 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.593623    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.593668 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.593664    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnls4\" (UniqueName: \"kubernetes.io/projected/896758a8-d6d7-4a40-8379-da91bd3603fc-kube-api-access-tnls4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.593903 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.593749    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.593903 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.593830    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/896758a8-d6d7-4a40-8379-da91bd3603fc-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.594033 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.594003    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.594099 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.594082    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.694799 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.694710    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.694799 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.694759    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.694799 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.694787    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.695073 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.694808    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnls4\" (UniqueName: \"kubernetes.io/projected/896758a8-d6d7-4a40-8379-da91bd3603fc-kube-api-access-tnls4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.695073 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.694838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.695073 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.694959    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/896758a8-d6d7-4a40-8379-da91bd3603fc-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.695223 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.695137    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.695223 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.695171    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.695341 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.695310    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.697203 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.697184    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.697407 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.697386    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/896758a8-d6d7-4a40-8379-da91bd3603fc-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:18.702439 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:18.702417    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnls4\" (UniqueName: \"kubernetes.io/projected/896758a8-d6d7-4a40-8379-da91bd3603fc-kube-api-access-tnls4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:20.919097 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:20.919055    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:20.923849 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:20.923823    2569 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:21.019555 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:21.019517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:21.019740 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:21.019665    2569 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:21.022232 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:21.022200    2569 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") DevicePath \"csi-01e7d1e59909e242017b8d39295dd7db4f28fee4332177fd46e940fea6dc042c\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:21.109936 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:21.109899    2569 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4ece3d96f092e7290ccd867b9e9ab8f9752db83e65e9e43e089f057ec23de585/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:21.126460 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:21.126427    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:21.150468 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:21.150433    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:21.285185 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:21.285157    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 29 23:49:21.287459 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:49:21.287431    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896758a8_d6d7_4a40_8379_da91bd3603fc.slice/crio-ea81f0066648efd65a0606eb4cb7608f94e5c292192d657966758e9e73dc8935 WatchSource:0}: Error finding container ea81f0066648efd65a0606eb4cb7608f94e5c292192d657966758e9e73dc8935: Status 404 returned error can't find the container with id ea81f0066648efd65a0606eb4cb7608f94e5c292192d657966758e9e73dc8935
Jul 29 23:49:21.328094 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:21.327876    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"896758a8-d6d7-4a40-8379-da91bd3603fc","Type":"ContainerStarted","Data":"ea81f0066648efd65a0606eb4cb7608f94e5c292192d657966758e9e73dc8935"}
Jul 29 23:49:22.339069 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:22.339031    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"896758a8-d6d7-4a40-8379-da91bd3603fc","Type":"ContainerStarted","Data":"ec2fe13baa3b3a23aac8bb409aab26d1ccdf5f57bcc4d79dd4bdbc520b871aa8"}
Jul 29 23:49:22.356408 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:22.356348    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=4.356332029 podStartE2EDuration="4.356332029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 23:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:49:22.356023776 +0000 UTC m=+3769.361221203" watchObservedRunningTime="2026-07-29 23:49:22.356332029 +0000 UTC m=+3769.361529451"
Jul 29 23:49:23.151379 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:23.151347    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:31.151287 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:31.151241    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:49:31.152809 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:31.152774    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="896758a8-d6d7-4a40-8379-da91bd3603fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 29 23:49:41.151869 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:41.151823    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="896758a8-d6d7-4a40-8379-da91bd3603fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 29 23:49:51.151617 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:49:51.151576    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="896758a8-d6d7-4a40-8379-da91bd3603fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 29 23:50:01.151772 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:50:01.151706    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="896758a8-d6d7-4a40-8379-da91bd3603fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 29 23:50:11.151097 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:50:11.151052    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="896758a8-d6d7-4a40-8379-da91bd3603fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 29 23:50:21.151150 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:50:21.151095    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="896758a8-d6d7-4a40-8379-da91bd3603fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 29 23:50:31.150974 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:50:31.150921    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="896758a8-d6d7-4a40-8379-da91bd3603fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 29 23:50:41.151043 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:50:41.150987    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="896758a8-d6d7-4a40-8379-da91bd3603fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 29 23:50:51.151577 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:50:51.151528    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="896758a8-d6d7-4a40-8379-da91bd3603fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 29 23:51:01.161425 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:01.161342    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:51:01.169148 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:01.169116    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:51:15.058507 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:15.058455    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 29 23:51:30.081740 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.081680    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="896758a8-d6d7-4a40-8379-da91bd3603fc" containerName="main" containerID="cri-o://ec2fe13baa3b3a23aac8bb409aab26d1ccdf5f57bcc4d79dd4bdbc520b871aa8" gracePeriod=45
Jul 29 23:51:30.162183 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.162143    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="896758a8-d6d7-4a40-8379-da91bd3603fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 29 23:51:30.829974 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.829948    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:51:30.880299 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.880219    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnls4\" (UniqueName: \"kubernetes.io/projected/896758a8-d6d7-4a40-8379-da91bd3603fc-kube-api-access-tnls4\") pod \"896758a8-d6d7-4a40-8379-da91bd3603fc\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") "
Jul 29 23:51:30.880299 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.880269    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-home\") pod \"896758a8-d6d7-4a40-8379-da91bd3603fc\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") "
Jul 29 23:51:30.880561 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.880410    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") pod \"896758a8-d6d7-4a40-8379-da91bd3603fc\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") "
Jul 29 23:51:30.880561 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.880445    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/896758a8-d6d7-4a40-8379-da91bd3603fc-tls-certs\") pod \"896758a8-d6d7-4a40-8379-da91bd3603fc\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") "
Jul 29 23:51:30.880561 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.880470    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-tmp-dir\") pod \"896758a8-d6d7-4a40-8379-da91bd3603fc\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") "
Jul 29 23:51:30.880561 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.880529    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-model-cache\") pod \"896758a8-d6d7-4a40-8379-da91bd3603fc\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") "
Jul 29 23:51:30.880561 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.880553    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-dshm\") pod \"896758a8-d6d7-4a40-8379-da91bd3603fc\" (UID: \"896758a8-d6d7-4a40-8379-da91bd3603fc\") "
Jul 29 23:51:30.880832 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.880793    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "896758a8-d6d7-4a40-8379-da91bd3603fc" (UID: "896758a8-d6d7-4a40-8379-da91bd3603fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:51:30.881098 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.881063    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-home" (OuterVolumeSpecName: "home") pod "896758a8-d6d7-4a40-8379-da91bd3603fc" (UID: "896758a8-d6d7-4a40-8379-da91bd3603fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:51:30.882735 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.882712    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/896758a8-d6d7-4a40-8379-da91bd3603fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "896758a8-d6d7-4a40-8379-da91bd3603fc" (UID: "896758a8-d6d7-4a40-8379-da91bd3603fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:51:30.882951 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.882926    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-dshm" (OuterVolumeSpecName: "dshm") pod "896758a8-d6d7-4a40-8379-da91bd3603fc" (UID: "896758a8-d6d7-4a40-8379-da91bd3603fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:51:30.883040 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.882948    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896758a8-d6d7-4a40-8379-da91bd3603fc-kube-api-access-tnls4" (OuterVolumeSpecName: "kube-api-access-tnls4") pod "896758a8-d6d7-4a40-8379-da91bd3603fc" (UID: "896758a8-d6d7-4a40-8379-da91bd3603fc"). InnerVolumeSpecName "kube-api-access-tnls4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:51:30.883538 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.883519    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222" (OuterVolumeSpecName: "kserve-pvc-source") pod "896758a8-d6d7-4a40-8379-da91bd3603fc" (UID: "896758a8-d6d7-4a40-8379-da91bd3603fc"). InnerVolumeSpecName "pvc-380fae1b-9e30-418e-9470-637495923811". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 29 23:51:30.897393 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.897356    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "896758a8-d6d7-4a40-8379-da91bd3603fc" (UID: "896758a8-d6d7-4a40-8379-da91bd3603fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:51:30.944188 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.944158    2569 generic.go:356] "Generic (PLEG): container finished" podID="896758a8-d6d7-4a40-8379-da91bd3603fc" containerID="ec2fe13baa3b3a23aac8bb409aab26d1ccdf5f57bcc4d79dd4bdbc520b871aa8" exitCode=0
Jul 29 23:51:30.944348 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.944201    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 23:51:30.944348 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.944226    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"896758a8-d6d7-4a40-8379-da91bd3603fc","Type":"ContainerDied","Data":"ec2fe13baa3b3a23aac8bb409aab26d1ccdf5f57bcc4d79dd4bdbc520b871aa8"}
Jul 29 23:51:30.944348 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.944267    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"896758a8-d6d7-4a40-8379-da91bd3603fc","Type":"ContainerDied","Data":"ea81f0066648efd65a0606eb4cb7608f94e5c292192d657966758e9e73dc8935"}
Jul 29 23:51:30.944348 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.944290    2569 scope.go:117] "RemoveContainer" containerID="ec2fe13baa3b3a23aac8bb409aab26d1ccdf5f57bcc4d79dd4bdbc520b871aa8"
Jul 29 23:51:30.953930 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.953908    2569 scope.go:117] "RemoveContainer" containerID="ec2fe13baa3b3a23aac8bb409aab26d1ccdf5f57bcc4d79dd4bdbc520b871aa8"
Jul 29 23:51:30.954190 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:51:30.954171    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec2fe13baa3b3a23aac8bb409aab26d1ccdf5f57bcc4d79dd4bdbc520b871aa8\": container with ID starting with ec2fe13baa3b3a23aac8bb409aab26d1ccdf5f57bcc4d79dd4bdbc520b871aa8 not found: ID does not exist" containerID="ec2fe13baa3b3a23aac8bb409aab26d1ccdf5f57bcc4d79dd4bdbc520b871aa8"
Jul 29 23:51:30.954254 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.954197    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2fe13baa3b3a23aac8bb409aab26d1ccdf5f57bcc4d79dd4bdbc520b871aa8"} err="failed to get container status \"ec2fe13baa3b3a23aac8bb409aab26d1ccdf5f57bcc4d79dd4bdbc520b871aa8\": rpc error: code = NotFound desc = could not find container \"ec2fe13baa3b3a23aac8bb409aab26d1ccdf5f57bcc4d79dd4bdbc520b871aa8\": container with ID starting with ec2fe13baa3b3a23aac8bb409aab26d1ccdf5f57bcc4d79dd4bdbc520b871aa8 not found: ID does not exist"
Jul 29 23:51:30.965109 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.965076    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 29 23:51:30.966536 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.966517    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 29 23:51:30.981221 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.981193    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/896758a8-d6d7-4a40-8379-da91bd3603fc-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:51:30.981346 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.981224    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:51:30.981346 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.981237    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:51:30.981346 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.981257    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:51:30.981346 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.981269    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnls4\" (UniqueName: \"kubernetes.io/projected/896758a8-d6d7-4a40-8379-da91bd3603fc-kube-api-access-tnls4\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:51:30.981346 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.981281    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/896758a8-d6d7-4a40-8379-da91bd3603fc-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:51:30.981346 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:30.981317    2569 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") on node \"ip-10-0-135-98.ec2.internal\" "
Jul 29 23:51:31.009436 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:31.009413    2569 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-380fae1b-9e30-418e-9470-637495923811" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222") on node "ip-10-0-135-98.ec2.internal"
Jul 29 23:51:31.081750 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:31.081720    2569 reconciler_common.go:299] "Volume detached for volume \"pvc-380fae1b-9e30-418e-9470-637495923811\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b43cdabe85a73222\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"csi-01e7d1e59909e242017b8d39295dd7db4f28fee4332177fd46e940fea6dc042c\""
Jul 29 23:51:31.473982 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:31.473944    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="896758a8-d6d7-4a40-8379-da91bd3603fc" path="/var/lib/kubelet/pods/896758a8-d6d7-4a40-8379-da91bd3603fc/volumes"
Jul 29 23:51:33.728624 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:33.728599    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:51:33.745416 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:33.745389    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:51:39.892244 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:39.892205    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"]
Jul 29 23:51:39.895352 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:39.895323    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-8m5qp"]
Jul 29 23:51:41.473985 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:51:41.473951    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30e0f470-dbcb-43b1-a947-652831e32f74" path="/var/lib/kubelet/pods/30e0f470-dbcb-43b1-a947-652831e32f74/volumes"
Jul 29 23:52:19.007033 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.006997    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"]
Jul 29 23:52:19.007544 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.007301    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="896758a8-d6d7-4a40-8379-da91bd3603fc" containerName="main"
Jul 29 23:52:19.007544 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.007312    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="896758a8-d6d7-4a40-8379-da91bd3603fc" containerName="main"
Jul 29 23:52:19.007544 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.007375    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="896758a8-d6d7-4a40-8379-da91bd3603fc" containerName="main"
Jul 29 23:52:19.010598 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.010573    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.013117 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.013093    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 29 23:52:19.013117 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.013110    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 29 23:52:19.013295 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.013153    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-gc9wb\""
Jul 29 23:52:19.013397 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.013377    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 29 23:52:19.013530 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.013458    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 29 23:52:19.020519 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.020496    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"]
Jul 29 23:52:19.186552 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.186516    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvjhv\" (UniqueName: \"kubernetes.io/projected/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-kube-api-access-tvjhv\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.186746 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.186570    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-dshm\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.186746 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.186617    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-tmp-dir\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.186746 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.186643    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-tls-certs\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.186746 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.186719    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-home\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.186913 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.186753    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-model-cache\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.186913 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.186798    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-kserve-provision-location\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.287925 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.287835    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvjhv\" (UniqueName: \"kubernetes.io/projected/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-kube-api-access-tvjhv\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.287925 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.287880    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-dshm\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.288130 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.287977    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-tmp-dir\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.288130 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.287993    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-tls-certs\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.288130 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.288017    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-home\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.288130 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.288037    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-model-cache\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.288130 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.288078    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-kserve-provision-location\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.288537 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.288474    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-tmp-dir\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.288537 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.288530    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-home\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.288707 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.288576    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-model-cache\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.288707 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.288589    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-kserve-provision-location\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.290290 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.290269    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-dshm\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.290725 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.290706    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-tls-certs\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.295749 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.295725    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvjhv\" (UniqueName: \"kubernetes.io/projected/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-kube-api-access-tvjhv\") pod \"stop-feature-test-kserve-b86c84465-28rmp\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.322635 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.322598    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:19.463692 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:19.463552    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"]
Jul 29 23:52:19.465943 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:52:19.465902    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f0e6f6_1b10_47b2_bbc5_af4ac3f4c6ad.slice/crio-f48c4e5832bb0237f13edcdeb44f91239db6b1541d53700626c85eb761d3e60c WatchSource:0}: Error finding container f48c4e5832bb0237f13edcdeb44f91239db6b1541d53700626c85eb761d3e60c: Status 404 returned error can't find the container with id f48c4e5832bb0237f13edcdeb44f91239db6b1541d53700626c85eb761d3e60c
Jul 29 23:52:20.153498 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:20.153450    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" event={"ID":"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad","Type":"ContainerStarted","Data":"1f7a1916217d9d71dac6dd5626b901b94228d8451fe5eb263224561be0ba6993"}
Jul 29 23:52:20.154010 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:20.153530    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" event={"ID":"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad","Type":"ContainerStarted","Data":"f48c4e5832bb0237f13edcdeb44f91239db6b1541d53700626c85eb761d3e60c"}
Jul 29 23:52:24.171655 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:24.171616    2569 generic.go:356] "Generic (PLEG): container finished" podID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerID="1f7a1916217d9d71dac6dd5626b901b94228d8451fe5eb263224561be0ba6993" exitCode=0
Jul 29 23:52:24.172045 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:24.171683    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" event={"ID":"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad","Type":"ContainerDied","Data":"1f7a1916217d9d71dac6dd5626b901b94228d8451fe5eb263224561be0ba6993"}
Jul 29 23:52:25.177912 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:25.177877    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" event={"ID":"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad","Type":"ContainerStarted","Data":"f5618c18c869f96bf857b72ba0c1ab82d354b016e20fb7e1b1981534e229d70a"}
Jul 29 23:52:25.196913 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:25.196796    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" podStartSLOduration=2.7214812200000003 podStartE2EDuration="7.196773068s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.475291848s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:52:25.195539239 +0000 UTC m=+3952.200736661" watchObservedRunningTime="2026-07-29 23:52:25.196773068 +0000 UTC m=+3952.201970487"
Jul 29 23:52:25.323032 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:25.322991    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:29.323233 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:29.323192    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:52:29.324672 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:29.324641    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 29 23:52:39.323736 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:39.323682    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 29 23:52:49.323995 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:49.323943    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 29 23:52:59.323316 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:52:59.323262    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 29 23:53:09.323214 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:53:09.323159    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 29 23:53:19.323271 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:53:19.323223    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 29 23:53:29.323235 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:53:29.323184    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 29 23:53:39.323257 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:53:39.323207    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 29 23:53:49.323930 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:53:49.323880    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 29 23:53:59.323952 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:53:59.323856    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 29 23:54:09.323604 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:09.323563    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 29 23:54:19.333640 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:19.333606    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:54:19.341374 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:19.341334    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:54:34.767785 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:34.767744    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"]
Jul 29 23:54:49.790126 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:49.790082    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main" containerID="cri-o://f5618c18c869f96bf857b72ba0c1ab82d354b016e20fb7e1b1981534e229d70a" gracePeriod=45
Jul 29 23:54:50.334307 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.334273    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 29 23:54:50.543639 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.543608    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:54:50.615681 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.615590    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvjhv\" (UniqueName: \"kubernetes.io/projected/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-kube-api-access-tvjhv\") pod \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") "
Jul 29 23:54:50.615681 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.615630    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-home\") pod \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") "
Jul 29 23:54:50.615681 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.615664    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-kserve-provision-location\") pod \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") "
Jul 29 23:54:50.615990 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.615691    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-tls-certs\") pod \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") "
Jul 29 23:54:50.615990 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.615713    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-model-cache\") pod \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") "
Jul 29 23:54:50.615990 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.615759    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-dshm\") pod \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") "
Jul 29 23:54:50.615990 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.615810    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-tmp-dir\") pod \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\" (UID: \"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad\") "
Jul 29 23:54:50.616188 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.616042    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" (UID: "66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:54:50.616188 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.616149    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:54:50.616540 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.616461    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-home" (OuterVolumeSpecName: "home") pod "66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" (UID: "66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:54:50.618585 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.618546    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" (UID: "66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:54:50.618978 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.618953    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-dshm" (OuterVolumeSpecName: "dshm") pod "66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" (UID: "66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:54:50.618978 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.618963    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-kube-api-access-tvjhv" (OuterVolumeSpecName: "kube-api-access-tvjhv") pod "66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" (UID: "66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad"). InnerVolumeSpecName "kube-api-access-tvjhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:54:50.628262 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.628226    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" (UID: "66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:54:50.681932 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.681885    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" (UID: "66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:54:50.716791 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.716752    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:54:50.716791 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.716785    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:54:50.716791 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.716795    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvjhv\" (UniqueName: \"kubernetes.io/projected/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-kube-api-access-tvjhv\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:54:50.717034 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.716815    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:54:50.717034 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.716826    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-kserve-provision-location\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:54:50.717034 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.716833    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:54:50.804240 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.804208    2569 generic.go:356] "Generic (PLEG): container finished" podID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerID="f5618c18c869f96bf857b72ba0c1ab82d354b016e20fb7e1b1981534e229d70a" exitCode=0
Jul 29 23:54:50.804758 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.804266    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"
Jul 29 23:54:50.804758 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.804298    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" event={"ID":"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad","Type":"ContainerDied","Data":"f5618c18c869f96bf857b72ba0c1ab82d354b016e20fb7e1b1981534e229d70a"}
Jul 29 23:54:50.804758 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.804381    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp" event={"ID":"66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad","Type":"ContainerDied","Data":"f48c4e5832bb0237f13edcdeb44f91239db6b1541d53700626c85eb761d3e60c"}
Jul 29 23:54:50.804758 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.804416    2569 scope.go:117] "RemoveContainer" containerID="f5618c18c869f96bf857b72ba0c1ab82d354b016e20fb7e1b1981534e229d70a"
Jul 29 23:54:50.813324 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.813297    2569 scope.go:117] "RemoveContainer" containerID="1f7a1916217d9d71dac6dd5626b901b94228d8451fe5eb263224561be0ba6993"
Jul 29 23:54:50.828968 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.828931    2569 scope.go:117] "RemoveContainer" containerID="f5618c18c869f96bf857b72ba0c1ab82d354b016e20fb7e1b1981534e229d70a"
Jul 29 23:54:50.829364 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:54:50.829337    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5618c18c869f96bf857b72ba0c1ab82d354b016e20fb7e1b1981534e229d70a\": container with ID starting with f5618c18c869f96bf857b72ba0c1ab82d354b016e20fb7e1b1981534e229d70a not found: ID does not exist" containerID="f5618c18c869f96bf857b72ba0c1ab82d354b016e20fb7e1b1981534e229d70a"
Jul 29 23:54:50.829464 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.829376    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5618c18c869f96bf857b72ba0c1ab82d354b016e20fb7e1b1981534e229d70a"} err="failed to get container status \"f5618c18c869f96bf857b72ba0c1ab82d354b016e20fb7e1b1981534e229d70a\": rpc error: code = NotFound desc = could not find container \"f5618c18c869f96bf857b72ba0c1ab82d354b016e20fb7e1b1981534e229d70a\": container with ID starting with f5618c18c869f96bf857b72ba0c1ab82d354b016e20fb7e1b1981534e229d70a not found: ID does not exist"
Jul 29 23:54:50.829464 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.829403    2569 scope.go:117] "RemoveContainer" containerID="1f7a1916217d9d71dac6dd5626b901b94228d8451fe5eb263224561be0ba6993"
Jul 29 23:54:50.830014 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:54:50.829992    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7a1916217d9d71dac6dd5626b901b94228d8451fe5eb263224561be0ba6993\": container with ID starting with 1f7a1916217d9d71dac6dd5626b901b94228d8451fe5eb263224561be0ba6993 not found: ID does not exist" containerID="1f7a1916217d9d71dac6dd5626b901b94228d8451fe5eb263224561be0ba6993"
Jul 29 23:54:50.830082 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.830022    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7a1916217d9d71dac6dd5626b901b94228d8451fe5eb263224561be0ba6993"} err="failed to get container status \"1f7a1916217d9d71dac6dd5626b901b94228d8451fe5eb263224561be0ba6993\": rpc error: code = NotFound desc = could not find container \"1f7a1916217d9d71dac6dd5626b901b94228d8451fe5eb263224561be0ba6993\": container with ID starting with 1f7a1916217d9d71dac6dd5626b901b94228d8451fe5eb263224561be0ba6993 not found: ID does not exist"
Jul 29 23:54:50.830225 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.830203    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"]
Jul 29 23:54:50.835034 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:50.835007    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-28rmp"]
Jul 29 23:54:51.474098 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:54:51.474063    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" path="/var/lib/kubelet/pods/66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad/volumes"
Jul 29 23:55:14.211043 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.211003    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"]
Jul 29 23:55:14.211505 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.211333    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="storage-initializer"
Jul 29 23:55:14.211505 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.211352    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="storage-initializer"
Jul 29 23:55:14.211505 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.211365    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main"
Jul 29 23:55:14.211505 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.211375    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main"
Jul 29 23:55:14.211505 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.211464    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="66f0e6f6-1b10-47b2-bbc5-af4ac3f4c6ad" containerName="main"
Jul 29 23:55:14.214461 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.214444    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.216556 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.216526    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-gc9wb\""
Jul 29 23:55:14.216687 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.216534    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 29 23:55:14.216687 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.216565    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 29 23:55:14.216869 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.216850    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 29 23:55:14.217165 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.217151    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 29 23:55:14.225934 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.225907    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"]
Jul 29 23:55:14.314818 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.314777    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-home\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.314818 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.314821    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/770adc2d-9da2-4755-a18b-794f8fb4cdd9-tls-certs\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.315051 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.314847    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-dshm\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.315051 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.314935    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnhfk\" (UniqueName: \"kubernetes.io/projected/770adc2d-9da2-4755-a18b-794f8fb4cdd9-kube-api-access-wnhfk\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.315051 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.314980    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-tmp-dir\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.315051 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.315044    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-kserve-provision-location\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.315208 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.315081    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-model-cache\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.415966 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.415929    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-dshm\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.416154 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.415976    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnhfk\" (UniqueName: \"kubernetes.io/projected/770adc2d-9da2-4755-a18b-794f8fb4cdd9-kube-api-access-wnhfk\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.416154 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.416016    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-tmp-dir\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.416154 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.416046    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-kserve-provision-location\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.416295 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.416159    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-model-cache\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.416295 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.416240    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-home\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.416295 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.416281    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/770adc2d-9da2-4755-a18b-794f8fb4cdd9-tls-certs\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.416504 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.416424    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-tmp-dir\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.416629 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.416608    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-kserve-provision-location\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.416724 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.416649    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-model-cache\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.416724 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.416677    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-home\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.418256 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.418238    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-dshm\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.418720 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.418704    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/770adc2d-9da2-4755-a18b-794f8fb4cdd9-tls-certs\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.423440 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.423420    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnhfk\" (UniqueName: \"kubernetes.io/projected/770adc2d-9da2-4755-a18b-794f8fb4cdd9-kube-api-access-wnhfk\") pod \"stop-feature-test-kserve-b86c84465-rtjjn\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.524980 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.524935    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:14.668715 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.668636    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"]
Jul 29 23:55:14.671066 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:55:14.671028    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770adc2d_9da2_4755_a18b_794f8fb4cdd9.slice/crio-a63818eb2332ee9a4035e8bbde47d30b6df077f528b36a6349e10359f383c4aa WatchSource:0}: Error finding container a63818eb2332ee9a4035e8bbde47d30b6df077f528b36a6349e10359f383c4aa: Status 404 returned error can't find the container with id a63818eb2332ee9a4035e8bbde47d30b6df077f528b36a6349e10359f383c4aa
Jul 29 23:55:14.908958 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.908850    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" event={"ID":"770adc2d-9da2-4755-a18b-794f8fb4cdd9","Type":"ContainerStarted","Data":"2ac27de580ac9938d5567ff2b4b9f6ffb654cfe79b788e5bf78cc0866b97c796"}
Jul 29 23:55:14.908958 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:14.908920    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" event={"ID":"770adc2d-9da2-4755-a18b-794f8fb4cdd9","Type":"ContainerStarted","Data":"a63818eb2332ee9a4035e8bbde47d30b6df077f528b36a6349e10359f383c4aa"}
Jul 29 23:55:21.944611 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:21.944512    2569 generic.go:356] "Generic (PLEG): container finished" podID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerID="2ac27de580ac9938d5567ff2b4b9f6ffb654cfe79b788e5bf78cc0866b97c796" exitCode=0
Jul 29 23:55:21.944611 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:21.944585    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" event={"ID":"770adc2d-9da2-4755-a18b-794f8fb4cdd9","Type":"ContainerDied","Data":"2ac27de580ac9938d5567ff2b4b9f6ffb654cfe79b788e5bf78cc0866b97c796"}
Jul 29 23:55:22.951174 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:22.951132    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" event={"ID":"770adc2d-9da2-4755-a18b-794f8fb4cdd9","Type":"ContainerStarted","Data":"0bd5437acd0476fbeda6f17721e7147bc3a564fe2079163cc72126eb2c05003d"}
Jul 29 23:55:22.971234 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:22.971177    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" podStartSLOduration=2.099172909 podStartE2EDuration="8.971161852s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.871988943s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:55:22.970002357 +0000 UTC m=+4129.975199780" watchObservedRunningTime="2026-07-29 23:55:22.971161852 +0000 UTC m=+4129.976359274"
Jul 29 23:55:23.526128 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:23.526066    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:24.525842 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:24.525809    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:55:24.527159 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:24.527129    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 29 23:55:33.822714 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:33.822621    2569 scope.go:117] "RemoveContainer" containerID="7a3d8eda4e1e73efd255255224695dafddb394424148b922b43d11a336e62113"
Jul 29 23:55:34.525850 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:34.525805    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 29 23:55:44.526105 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:44.526049    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 29 23:55:54.526092 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:55:54.526048    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 29 23:56:04.525636 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:56:04.525582    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 29 23:56:14.526220 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:56:14.526172    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 29 23:56:24.526144 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:56:24.526096    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 29 23:56:33.762895 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:56:33.762777    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:56:33.772388 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:56:33.772366    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 29 23:56:34.525648 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:56:34.525605    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 29 23:56:44.525700 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:56:44.525649    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 29 23:56:54.525388 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:56:54.525346    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 29 23:57:04.526469 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:04.526363    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 29 23:57:14.525943 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:14.525888    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 29 23:57:24.535844 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:24.535807    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:57:24.543727 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:24.543700    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:57:36.644140 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:36.644105    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"]
Jul 29 23:57:51.669129 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:51.669083    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main" containerID="cri-o://0bd5437acd0476fbeda6f17721e7147bc3a564fe2079163cc72126eb2c05003d" gracePeriod=45
Jul 29 23:57:52.428986 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.428920    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:57:52.596425 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.596391    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-home\") pod \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") "
Jul 29 23:57:52.596637 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.596438    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-tmp-dir\") pod \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") "
Jul 29 23:57:52.596637 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.596465    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnhfk\" (UniqueName: \"kubernetes.io/projected/770adc2d-9da2-4755-a18b-794f8fb4cdd9-kube-api-access-wnhfk\") pod \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") "
Jul 29 23:57:52.596637 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.596503    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-dshm\") pod \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") "
Jul 29 23:57:52.596637 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.596531    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-kserve-provision-location\") pod \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") "
Jul 29 23:57:52.596637 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.596565    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/770adc2d-9da2-4755-a18b-794f8fb4cdd9-tls-certs\") pod \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") "
Jul 29 23:57:52.596914 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.596646    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-model-cache\") pod \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\" (UID: \"770adc2d-9da2-4755-a18b-794f8fb4cdd9\") "
Jul 29 23:57:52.597089 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.597056    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-model-cache" (OuterVolumeSpecName: "model-cache") pod "770adc2d-9da2-4755-a18b-794f8fb4cdd9" (UID: "770adc2d-9da2-4755-a18b-794f8fb4cdd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:57:52.597223 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.597194    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-home" (OuterVolumeSpecName: "home") pod "770adc2d-9da2-4755-a18b-794f8fb4cdd9" (UID: "770adc2d-9da2-4755-a18b-794f8fb4cdd9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:57:52.599540 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.599513    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770adc2d-9da2-4755-a18b-794f8fb4cdd9-kube-api-access-wnhfk" (OuterVolumeSpecName: "kube-api-access-wnhfk") pod "770adc2d-9da2-4755-a18b-794f8fb4cdd9" (UID: "770adc2d-9da2-4755-a18b-794f8fb4cdd9"). InnerVolumeSpecName "kube-api-access-wnhfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:57:52.599666 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.599581    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-dshm" (OuterVolumeSpecName: "dshm") pod "770adc2d-9da2-4755-a18b-794f8fb4cdd9" (UID: "770adc2d-9da2-4755-a18b-794f8fb4cdd9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:57:52.599794 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.599768    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/770adc2d-9da2-4755-a18b-794f8fb4cdd9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "770adc2d-9da2-4755-a18b-794f8fb4cdd9" (UID: "770adc2d-9da2-4755-a18b-794f8fb4cdd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:57:52.599929 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.599875    2569 generic.go:356] "Generic (PLEG): container finished" podID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerID="0bd5437acd0476fbeda6f17721e7147bc3a564fe2079163cc72126eb2c05003d" exitCode=0
Jul 29 23:57:52.599994 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.599958    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"
Jul 29 23:57:52.599994 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.599947    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" event={"ID":"770adc2d-9da2-4755-a18b-794f8fb4cdd9","Type":"ContainerDied","Data":"0bd5437acd0476fbeda6f17721e7147bc3a564fe2079163cc72126eb2c05003d"}
Jul 29 23:57:52.600098 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.600019    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn" event={"ID":"770adc2d-9da2-4755-a18b-794f8fb4cdd9","Type":"ContainerDied","Data":"a63818eb2332ee9a4035e8bbde47d30b6df077f528b36a6349e10359f383c4aa"}
Jul 29 23:57:52.600098 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.600053    2569 scope.go:117] "RemoveContainer" containerID="0bd5437acd0476fbeda6f17721e7147bc3a564fe2079163cc72126eb2c05003d"
Jul 29 23:57:52.610972 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.610917    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "770adc2d-9da2-4755-a18b-794f8fb4cdd9" (UID: "770adc2d-9da2-4755-a18b-794f8fb4cdd9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:57:52.621864 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.621841    2569 scope.go:117] "RemoveContainer" containerID="2ac27de580ac9938d5567ff2b4b9f6ffb654cfe79b788e5bf78cc0866b97c796"
Jul 29 23:57:52.631705 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.631682    2569 scope.go:117] "RemoveContainer" containerID="0bd5437acd0476fbeda6f17721e7147bc3a564fe2079163cc72126eb2c05003d"
Jul 29 23:57:52.632000 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:57:52.631976    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd5437acd0476fbeda6f17721e7147bc3a564fe2079163cc72126eb2c05003d\": container with ID starting with 0bd5437acd0476fbeda6f17721e7147bc3a564fe2079163cc72126eb2c05003d not found: ID does not exist" containerID="0bd5437acd0476fbeda6f17721e7147bc3a564fe2079163cc72126eb2c05003d"
Jul 29 23:57:52.632098 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.632010    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd5437acd0476fbeda6f17721e7147bc3a564fe2079163cc72126eb2c05003d"} err="failed to get container status \"0bd5437acd0476fbeda6f17721e7147bc3a564fe2079163cc72126eb2c05003d\": rpc error: code = NotFound desc = could not find container \"0bd5437acd0476fbeda6f17721e7147bc3a564fe2079163cc72126eb2c05003d\": container with ID starting with 0bd5437acd0476fbeda6f17721e7147bc3a564fe2079163cc72126eb2c05003d not found: ID does not exist"
Jul 29 23:57:52.632098 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.632033    2569 scope.go:117] "RemoveContainer" containerID="2ac27de580ac9938d5567ff2b4b9f6ffb654cfe79b788e5bf78cc0866b97c796"
Jul 29 23:57:52.632288 ip-10-0-135-98 kubenswrapper[2569]: E0729 23:57:52.632272    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ac27de580ac9938d5567ff2b4b9f6ffb654cfe79b788e5bf78cc0866b97c796\": container with ID starting with 2ac27de580ac9938d5567ff2b4b9f6ffb654cfe79b788e5bf78cc0866b97c796 not found: ID does not exist" containerID="2ac27de580ac9938d5567ff2b4b9f6ffb654cfe79b788e5bf78cc0866b97c796"
Jul 29 23:57:52.632348 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.632292    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ac27de580ac9938d5567ff2b4b9f6ffb654cfe79b788e5bf78cc0866b97c796"} err="failed to get container status \"2ac27de580ac9938d5567ff2b4b9f6ffb654cfe79b788e5bf78cc0866b97c796\": rpc error: code = NotFound desc = could not find container \"2ac27de580ac9938d5567ff2b4b9f6ffb654cfe79b788e5bf78cc0866b97c796\": container with ID starting with 2ac27de580ac9938d5567ff2b4b9f6ffb654cfe79b788e5bf78cc0866b97c796 not found: ID does not exist"
Jul 29 23:57:52.670212 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.670165    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "770adc2d-9da2-4755-a18b-794f8fb4cdd9" (UID: "770adc2d-9da2-4755-a18b-794f8fb4cdd9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:57:52.697528 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.697435    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:57:52.697528 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.697462    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:57:52.697528 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.697471    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnhfk\" (UniqueName: \"kubernetes.io/projected/770adc2d-9da2-4755-a18b-794f8fb4cdd9-kube-api-access-wnhfk\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:57:52.697528 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.697491    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:57:52.697528 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.697501    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-kserve-provision-location\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:57:52.697528 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.697512    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/770adc2d-9da2-4755-a18b-794f8fb4cdd9-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:57:52.697528 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.697520    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/770adc2d-9da2-4755-a18b-794f8fb4cdd9-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 29 23:57:52.921330 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.921295    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"]
Jul 29 23:57:52.924760 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:52.924729    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b86c84465-rtjjn"]
Jul 29 23:57:53.473900 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:57:53.473863    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" path="/var/lib/kubelet/pods/770adc2d-9da2-4755-a18b-794f8fb4cdd9/volumes"
Jul 29 23:58:14.987991 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:14.987948    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"]
Jul 29 23:58:14.988559 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:14.988394    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="storage-initializer"
Jul 29 23:58:14.988559 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:14.988413    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="storage-initializer"
Jul 29 23:58:14.988559 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:14.988436    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main"
Jul 29 23:58:14.988559 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:14.988445    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main"
Jul 29 23:58:14.988559 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:14.988553    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="770adc2d-9da2-4755-a18b-794f8fb4cdd9" containerName="main"
Jul 29 23:58:14.991440 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:14.991414    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:14.993492 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:14.993455    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Jul 29 23:58:14.993772 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:14.993748    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 23:58:14.993772 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:14.993765    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-t6l56\""
Jul 29 23:58:14.993971 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:14.993817    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Jul 29 23:58:14.994076 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:14.993998    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Jul 29 23:58:14.994149 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:14.994075    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 29 23:58:15.005026 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.004996    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"]
Jul 29 23:58:15.053695 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.053661    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-dshm\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.053865 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.053702    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r472\" (UniqueName: \"kubernetes.io/projected/a7242411-8e01-49fc-9b2d-8f9ae6672858-kube-api-access-8r472\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.053865 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.053735    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7242411-8e01-49fc-9b2d-8f9ae6672858-cabundle-cert\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.053865 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.053785    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-tmp-dir\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.053978 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.053863    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-home\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.053978 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.053906    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-model-cache\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.053978 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.053936    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7242411-8e01-49fc-9b2d-8f9ae6672858-tls-certs\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.053978 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.053952    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.155329 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.155295    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-model-cache\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.155538 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.155338    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7242411-8e01-49fc-9b2d-8f9ae6672858-tls-certs\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.155538 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.155358    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.155538 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.155407    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-dshm\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.155538 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.155432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r472\" (UniqueName: \"kubernetes.io/projected/a7242411-8e01-49fc-9b2d-8f9ae6672858-kube-api-access-8r472\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.155538 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.155466    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7242411-8e01-49fc-9b2d-8f9ae6672858-cabundle-cert\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.155538 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.155516    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-tmp-dir\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.155881 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.155565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-home\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.155881 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.155770    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-model-cache\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.155881 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.155833    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.156043 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.155914    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-home\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.156131 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.156098    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-tmp-dir\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.156526 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.156476    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7242411-8e01-49fc-9b2d-8f9ae6672858-cabundle-cert\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.157741 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.157719    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-dshm\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.158061 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.158044    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7242411-8e01-49fc-9b2d-8f9ae6672858-tls-certs\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.162787 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.162766    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r472\" (UniqueName: \"kubernetes.io/projected/a7242411-8e01-49fc-9b2d-8f9ae6672858-kube-api-access-8r472\") pod \"lora-single-adapter-test-kserve-687ffc9ffd-srldp\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.304195 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.304096    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:15.443679 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.443560    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"]
Jul 29 23:58:15.446447 ip-10-0-135-98 kubenswrapper[2569]: W0729 23:58:15.446419    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7242411_8e01_49fc_9b2d_8f9ae6672858.slice/crio-0e6abddd5c8bd29be331311b3dc97c498e189ca11af4160a585ea8a89e11e574 WatchSource:0}: Error finding container 0e6abddd5c8bd29be331311b3dc97c498e189ca11af4160a585ea8a89e11e574: Status 404 returned error can't find the container with id 0e6abddd5c8bd29be331311b3dc97c498e189ca11af4160a585ea8a89e11e574
Jul 29 23:58:15.697138 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.697036    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" event={"ID":"a7242411-8e01-49fc-9b2d-8f9ae6672858","Type":"ContainerStarted","Data":"aabd8e86a825024c7fe9dc90fdb55a564b3831f124b69f3e29f36e35b5d30e3e"}
Jul 29 23:58:15.697138 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:15.697100    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" event={"ID":"a7242411-8e01-49fc-9b2d-8f9ae6672858","Type":"ContainerStarted","Data":"0e6abddd5c8bd29be331311b3dc97c498e189ca11af4160a585ea8a89e11e574"}
Jul 29 23:58:22.728063 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:22.728026    2569 generic.go:356] "Generic (PLEG): container finished" podID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerID="aabd8e86a825024c7fe9dc90fdb55a564b3831f124b69f3e29f36e35b5d30e3e" exitCode=0
Jul 29 23:58:22.728532 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:22.728095    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" event={"ID":"a7242411-8e01-49fc-9b2d-8f9ae6672858","Type":"ContainerDied","Data":"aabd8e86a825024c7fe9dc90fdb55a564b3831f124b69f3e29f36e35b5d30e3e"}
Jul 29 23:58:23.735275 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:23.735234    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" event={"ID":"a7242411-8e01-49fc-9b2d-8f9ae6672858","Type":"ContainerStarted","Data":"2f4b329644aeca4e0ef33e98567e15544f3634de809da1736325863b1bce2306"}
Jul 29 23:58:23.754660 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:23.754599    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" podStartSLOduration=3.343353141 podStartE2EDuration="9.754583861s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.41123072s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:58:23.753562325 +0000 UTC m=+4310.758759749" watchObservedRunningTime="2026-07-29 23:58:23.754583861 +0000 UTC m=+4310.759781283"
Jul 29 23:58:24.304848 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:24.304801    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:25.304686 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:25.304596    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:58:25.306225 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:25.306189    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 29 23:58:35.304972 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:35.304913    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 29 23:58:45.304619 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:45.304577    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 29 23:58:55.304987 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:58:55.304943    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 29 23:59:05.305165 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:59:05.305114    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 29 23:59:15.304533 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:59:15.304463    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 29 23:59:25.304843 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:59:25.304788    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 29 23:59:35.304714 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:59:35.304668    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 29 23:59:45.315048 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:59:45.315016    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 29 23:59:45.322797 ip-10-0-135-98 kubenswrapper[2569]: I0729 23:59:45.322770    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 30 00:00:29.837934 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:29.837904    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-687ffc9ffd-srldp_a7242411-8e01-49fc-9b2d-8f9ae6672858/storage-initializer/0.log"
Jul 30 00:00:29.846369 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:29.846341    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-687ffc9ffd-srldp_a7242411-8e01-49fc-9b2d-8f9ae6672858/main/0.log"
Jul 30 00:00:33.216834 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:33.216799    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"]
Jul 30 00:00:48.241597 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.241545    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerName="main" containerID="cri-o://2f4b329644aeca4e0ef33e98567e15544f3634de809da1736325863b1bce2306" gracePeriod=45
Jul 30 00:00:48.315752 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.315707    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 30 00:00:48.913367 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.913333    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 30 00:00:48.919693 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.919661    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-model-cache\") pod \"a7242411-8e01-49fc-9b2d-8f9ae6672858\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") "
Jul 30 00:00:48.919825 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.919722    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r472\" (UniqueName: \"kubernetes.io/projected/a7242411-8e01-49fc-9b2d-8f9ae6672858-kube-api-access-8r472\") pod \"a7242411-8e01-49fc-9b2d-8f9ae6672858\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") "
Jul 30 00:00:48.919825 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.919775    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-kserve-provision-location\") pod \"a7242411-8e01-49fc-9b2d-8f9ae6672858\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") "
Jul 30 00:00:48.919945 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.919836    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-tmp-dir\") pod \"a7242411-8e01-49fc-9b2d-8f9ae6672858\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") "
Jul 30 00:00:48.919945 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.919872    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-dshm\") pod \"a7242411-8e01-49fc-9b2d-8f9ae6672858\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") "
Jul 30 00:00:48.919945 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.919921    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-home\") pod \"a7242411-8e01-49fc-9b2d-8f9ae6672858\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") "
Jul 30 00:00:48.920110 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.919943    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7242411-8e01-49fc-9b2d-8f9ae6672858" (UID: "a7242411-8e01-49fc-9b2d-8f9ae6672858"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:00:48.920110 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.919947    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7242411-8e01-49fc-9b2d-8f9ae6672858-tls-certs\") pod \"a7242411-8e01-49fc-9b2d-8f9ae6672858\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") "
Jul 30 00:00:48.920110 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.919992    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7242411-8e01-49fc-9b2d-8f9ae6672858-cabundle-cert\") pod \"a7242411-8e01-49fc-9b2d-8f9ae6672858\" (UID: \"a7242411-8e01-49fc-9b2d-8f9ae6672858\") "
Jul 30 00:00:48.920346 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.920324    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:00:48.920539 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.920509    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-home" (OuterVolumeSpecName: "home") pod "a7242411-8e01-49fc-9b2d-8f9ae6672858" (UID: "a7242411-8e01-49fc-9b2d-8f9ae6672858"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:00:48.920756 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.920724    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7242411-8e01-49fc-9b2d-8f9ae6672858-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a7242411-8e01-49fc-9b2d-8f9ae6672858" (UID: "a7242411-8e01-49fc-9b2d-8f9ae6672858"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:00:48.922359 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.922293    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-dshm" (OuterVolumeSpecName: "dshm") pod "a7242411-8e01-49fc-9b2d-8f9ae6672858" (UID: "a7242411-8e01-49fc-9b2d-8f9ae6672858"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:00:48.922472 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.922360    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7242411-8e01-49fc-9b2d-8f9ae6672858-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7242411-8e01-49fc-9b2d-8f9ae6672858" (UID: "a7242411-8e01-49fc-9b2d-8f9ae6672858"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:00:48.922879 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.922853    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7242411-8e01-49fc-9b2d-8f9ae6672858-kube-api-access-8r472" (OuterVolumeSpecName: "kube-api-access-8r472") pod "a7242411-8e01-49fc-9b2d-8f9ae6672858" (UID: "a7242411-8e01-49fc-9b2d-8f9ae6672858"). InnerVolumeSpecName "kube-api-access-8r472". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:00:48.936352 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.936292    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7242411-8e01-49fc-9b2d-8f9ae6672858" (UID: "a7242411-8e01-49fc-9b2d-8f9ae6672858"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:00:48.996805 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:48.996756    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7242411-8e01-49fc-9b2d-8f9ae6672858" (UID: "a7242411-8e01-49fc-9b2d-8f9ae6672858"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:00:49.021167 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.021129    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r472\" (UniqueName: \"kubernetes.io/projected/a7242411-8e01-49fc-9b2d-8f9ae6672858-kube-api-access-8r472\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:00:49.021167 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.021161    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-kserve-provision-location\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:00:49.021374 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.021175    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:00:49.021374 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.021187    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:00:49.021374 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.021198    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7242411-8e01-49fc-9b2d-8f9ae6672858-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:00:49.021374 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.021208    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7242411-8e01-49fc-9b2d-8f9ae6672858-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:00:49.021374 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.021219    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7242411-8e01-49fc-9b2d-8f9ae6672858-cabundle-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:00:49.398576 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.398539    2569 generic.go:356] "Generic (PLEG): container finished" podID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerID="2f4b329644aeca4e0ef33e98567e15544f3634de809da1736325863b1bce2306" exitCode=0
Jul 30 00:00:49.399048 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.398589    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"
Jul 30 00:00:49.399048 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.398616    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" event={"ID":"a7242411-8e01-49fc-9b2d-8f9ae6672858","Type":"ContainerDied","Data":"2f4b329644aeca4e0ef33e98567e15544f3634de809da1736325863b1bce2306"}
Jul 30 00:00:49.399048 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.398687    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp" event={"ID":"a7242411-8e01-49fc-9b2d-8f9ae6672858","Type":"ContainerDied","Data":"0e6abddd5c8bd29be331311b3dc97c498e189ca11af4160a585ea8a89e11e574"}
Jul 30 00:00:49.399048 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.398713    2569 scope.go:117] "RemoveContainer" containerID="2f4b329644aeca4e0ef33e98567e15544f3634de809da1736325863b1bce2306"
Jul 30 00:00:49.407727 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.407703    2569 scope.go:117] "RemoveContainer" containerID="aabd8e86a825024c7fe9dc90fdb55a564b3831f124b69f3e29f36e35b5d30e3e"
Jul 30 00:00:49.415572 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.415552    2569 scope.go:117] "RemoveContainer" containerID="2f4b329644aeca4e0ef33e98567e15544f3634de809da1736325863b1bce2306"
Jul 30 00:00:49.415819 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:00:49.415798    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4b329644aeca4e0ef33e98567e15544f3634de809da1736325863b1bce2306\": container with ID starting with 2f4b329644aeca4e0ef33e98567e15544f3634de809da1736325863b1bce2306 not found: ID does not exist" containerID="2f4b329644aeca4e0ef33e98567e15544f3634de809da1736325863b1bce2306"
Jul 30 00:00:49.415871 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.415827    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4b329644aeca4e0ef33e98567e15544f3634de809da1736325863b1bce2306"} err="failed to get container status \"2f4b329644aeca4e0ef33e98567e15544f3634de809da1736325863b1bce2306\": rpc error: code = NotFound desc = could not find container \"2f4b329644aeca4e0ef33e98567e15544f3634de809da1736325863b1bce2306\": container with ID starting with 2f4b329644aeca4e0ef33e98567e15544f3634de809da1736325863b1bce2306 not found: ID does not exist"
Jul 30 00:00:49.415871 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.415843    2569 scope.go:117] "RemoveContainer" containerID="aabd8e86a825024c7fe9dc90fdb55a564b3831f124b69f3e29f36e35b5d30e3e"
Jul 30 00:00:49.416067 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:00:49.416050    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabd8e86a825024c7fe9dc90fdb55a564b3831f124b69f3e29f36e35b5d30e3e\": container with ID starting with aabd8e86a825024c7fe9dc90fdb55a564b3831f124b69f3e29f36e35b5d30e3e not found: ID does not exist" containerID="aabd8e86a825024c7fe9dc90fdb55a564b3831f124b69f3e29f36e35b5d30e3e"
Jul 30 00:00:49.416109 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.416071    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabd8e86a825024c7fe9dc90fdb55a564b3831f124b69f3e29f36e35b5d30e3e"} err="failed to get container status \"aabd8e86a825024c7fe9dc90fdb55a564b3831f124b69f3e29f36e35b5d30e3e\": rpc error: code = NotFound desc = could not find container \"aabd8e86a825024c7fe9dc90fdb55a564b3831f124b69f3e29f36e35b5d30e3e\": container with ID starting with aabd8e86a825024c7fe9dc90fdb55a564b3831f124b69f3e29f36e35b5d30e3e not found: ID does not exist"
Jul 30 00:00:49.420705 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.420680    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"]
Jul 30 00:00:49.423948 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.423928    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-687ffc9ffd-srldp"]
Jul 30 00:00:49.474286 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:00:49.474256    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" path="/var/lib/kubelet/pods/a7242411-8e01-49fc-9b2d-8f9ae6672858/volumes"
Jul 30 00:01:32.205521 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.205472    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"]
Jul 30 00:01:32.206016 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.205956    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerName="storage-initializer"
Jul 30 00:01:32.206016 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.205976    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerName="storage-initializer"
Jul 30 00:01:32.206016 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.205995    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerName="main"
Jul 30 00:01:32.206016 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.206004    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerName="main"
Jul 30 00:01:32.206151 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.206084    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7242411-8e01-49fc-9b2d-8f9ae6672858" containerName="main"
Jul 30 00:01:32.209439 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.209413    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.211569 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.211538    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Jul 30 00:01:32.211742 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.211578    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 00:01:32.211742 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.211631    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-ttv88\""
Jul 30 00:01:32.211742 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.211685    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 30 00:01:32.211742 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.211699    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Jul 30 00:01:32.211742 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.211727    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Jul 30 00:01:32.224108 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.224078    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"]
Jul 30 00:01:32.401026 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.400986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ef8448-4b9b-4260-957a-802fedba2e83-tls-certs\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.401241 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.401039    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55ef8448-4b9b-4260-957a-802fedba2e83-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.401241 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.401159    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-home\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.401241 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.401232    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-model-cache\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.401420 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.401277    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.401420 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.401342    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.401420 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.401369    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdzcq\" (UniqueName: \"kubernetes.io/projected/55ef8448-4b9b-4260-957a-802fedba2e83-kube-api-access-hdzcq\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.401420 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.401406    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-dshm\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.502786 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.502743    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.502998 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.502790    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdzcq\" (UniqueName: \"kubernetes.io/projected/55ef8448-4b9b-4260-957a-802fedba2e83-kube-api-access-hdzcq\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.502998 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.502827    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-dshm\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.502998 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.502857    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ef8448-4b9b-4260-957a-802fedba2e83-tls-certs\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.502998 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.502890    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55ef8448-4b9b-4260-957a-802fedba2e83-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.502998 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.502935    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-home\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.502998 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.502982    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-model-cache\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.503333 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.503023    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.503333 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.503226    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.503454 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.503333    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-model-cache\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.503454 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.503355    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-home\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.503454 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.503440    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.503605 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.503575    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55ef8448-4b9b-4260-957a-802fedba2e83-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.505378 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.505353    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-dshm\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.505474 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.505452    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ef8448-4b9b-4260-957a-802fedba2e83-tls-certs\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.510652 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.510631    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdzcq\" (UniqueName: \"kubernetes.io/projected/55ef8448-4b9b-4260-957a-802fedba2e83-kube-api-access-hdzcq\") pod \"lora-multiple-adapters-test-kserve-689f8679cf-fm9ss\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.520264 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.520237    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:32.659867 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:32.659834    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"]
Jul 30 00:01:32.662373 ip-10-0-135-98 kubenswrapper[2569]: W0730 00:01:32.662334    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ef8448_4b9b_4260_957a_802fedba2e83.slice/crio-d90423c53c40a32eb7c06f6196e1a7861ee4dbb4b9b9fa1661f13bd0e9a59667 WatchSource:0}: Error finding container d90423c53c40a32eb7c06f6196e1a7861ee4dbb4b9b9fa1661f13bd0e9a59667: Status 404 returned error can't find the container with id d90423c53c40a32eb7c06f6196e1a7861ee4dbb4b9b9fa1661f13bd0e9a59667
Jul 30 00:01:33.578159 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:33.578119    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" event={"ID":"55ef8448-4b9b-4260-957a-802fedba2e83","Type":"ContainerStarted","Data":"25e16d15577a0a51ba93fdfa6ba9acff5f87dd868f8ac790bc9afda5b52f11cf"}
Jul 30 00:01:33.578579 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:33.578174    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" event={"ID":"55ef8448-4b9b-4260-957a-802fedba2e83","Type":"ContainerStarted","Data":"d90423c53c40a32eb7c06f6196e1a7861ee4dbb4b9b9fa1661f13bd0e9a59667"}
Jul 30 00:01:33.794290 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:33.794191    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 30 00:01:33.810127 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:33.804566    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 30 00:01:39.603132 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:39.603093    2569 generic.go:356] "Generic (PLEG): container finished" podID="55ef8448-4b9b-4260-957a-802fedba2e83" containerID="25e16d15577a0a51ba93fdfa6ba9acff5f87dd868f8ac790bc9afda5b52f11cf" exitCode=0
Jul 30 00:01:39.603551 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:39.603165    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" event={"ID":"55ef8448-4b9b-4260-957a-802fedba2e83","Type":"ContainerDied","Data":"25e16d15577a0a51ba93fdfa6ba9acff5f87dd868f8ac790bc9afda5b52f11cf"}
Jul 30 00:01:40.609972 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:40.609935    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" event={"ID":"55ef8448-4b9b-4260-957a-802fedba2e83","Type":"ContainerStarted","Data":"33a722d3e070efb89a52bfa148183d25997c16db184eb3a6fea6ab9001406c76"}
Jul 30 00:01:40.631814 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:40.631752    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" podStartSLOduration=2.382427129 podStartE2EDuration="8.63171328s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.249286151s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:01:40.628801536 +0000 UTC m=+4507.633998983" watchObservedRunningTime="2026-07-30 00:01:40.63171328 +0000 UTC m=+4507.636910704"
Jul 30 00:01:41.520782 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:41.520737    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:42.520437 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:42.520396    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:01:42.522057 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:42.522018    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" podUID="55ef8448-4b9b-4260-957a-802fedba2e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 30 00:01:52.520742 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:01:52.520689    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" podUID="55ef8448-4b9b-4260-957a-802fedba2e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 30 00:02:02.521020 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:02:02.520969    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" podUID="55ef8448-4b9b-4260-957a-802fedba2e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 30 00:02:12.521031 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:02:12.520976    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" podUID="55ef8448-4b9b-4260-957a-802fedba2e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 30 00:02:22.521434 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:02:22.521381    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" podUID="55ef8448-4b9b-4260-957a-802fedba2e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 30 00:02:32.521674 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:02:32.521627    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" podUID="55ef8448-4b9b-4260-957a-802fedba2e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 30 00:02:42.521363 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:02:42.521303    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" podUID="55ef8448-4b9b-4260-957a-802fedba2e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 30 00:02:52.521110 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:02:52.521059    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" podUID="55ef8448-4b9b-4260-957a-802fedba2e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 30 00:03:02.531285 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:03:02.531186    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:03:02.539264 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:03:02.539230    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:03:59.994718 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:03:59.994685    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-689f8679cf-fm9ss_55ef8448-4b9b-4260-957a-802fedba2e83/storage-initializer/0.log"
Jul 30 00:04:00.003955 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:00.003930    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-689f8679cf-fm9ss_55ef8448-4b9b-4260-957a-802fedba2e83/main/0.log"
Jul 30 00:04:09.498610 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:09.498566    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"]
Jul 30 00:04:24.521573 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:24.521518    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" podUID="55ef8448-4b9b-4260-957a-802fedba2e83" containerName="main" containerID="cri-o://33a722d3e070efb89a52bfa148183d25997c16db184eb3a6fea6ab9001406c76" gracePeriod=45
Jul 30 00:04:25.181160 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.181130    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:04:25.246518 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.246365    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55ef8448-4b9b-4260-957a-802fedba2e83-cabundle-cert\") pod \"55ef8448-4b9b-4260-957a-802fedba2e83\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") "
Jul 30 00:04:25.246518 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.246424    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdzcq\" (UniqueName: \"kubernetes.io/projected/55ef8448-4b9b-4260-957a-802fedba2e83-kube-api-access-hdzcq\") pod \"55ef8448-4b9b-4260-957a-802fedba2e83\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") "
Jul 30 00:04:25.246518 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.246457    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-model-cache\") pod \"55ef8448-4b9b-4260-957a-802fedba2e83\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") "
Jul 30 00:04:25.246518 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.246477    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-dshm\") pod \"55ef8448-4b9b-4260-957a-802fedba2e83\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") "
Jul 30 00:04:25.246874 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.246533    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-kserve-provision-location\") pod \"55ef8448-4b9b-4260-957a-802fedba2e83\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") "
Jul 30 00:04:25.246874 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.246626    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-home\") pod \"55ef8448-4b9b-4260-957a-802fedba2e83\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") "
Jul 30 00:04:25.246874 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.246660    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ef8448-4b9b-4260-957a-802fedba2e83-tls-certs\") pod \"55ef8448-4b9b-4260-957a-802fedba2e83\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") "
Jul 30 00:04:25.246874 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.246682    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-tmp-dir\") pod \"55ef8448-4b9b-4260-957a-802fedba2e83\" (UID: \"55ef8448-4b9b-4260-957a-802fedba2e83\") "
Jul 30 00:04:25.246874 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.246770    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-model-cache" (OuterVolumeSpecName: "model-cache") pod "55ef8448-4b9b-4260-957a-802fedba2e83" (UID: "55ef8448-4b9b-4260-957a-802fedba2e83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:04:25.247210 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.246871    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ef8448-4b9b-4260-957a-802fedba2e83-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "55ef8448-4b9b-4260-957a-802fedba2e83" (UID: "55ef8448-4b9b-4260-957a-802fedba2e83"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:04:25.247210 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.246979    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:04:25.247210 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.247040    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-home" (OuterVolumeSpecName: "home") pod "55ef8448-4b9b-4260-957a-802fedba2e83" (UID: "55ef8448-4b9b-4260-957a-802fedba2e83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:04:25.248836 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.248809    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-dshm" (OuterVolumeSpecName: "dshm") pod "55ef8448-4b9b-4260-957a-802fedba2e83" (UID: "55ef8448-4b9b-4260-957a-802fedba2e83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:04:25.249105 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.249078    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ef8448-4b9b-4260-957a-802fedba2e83-kube-api-access-hdzcq" (OuterVolumeSpecName: "kube-api-access-hdzcq") pod "55ef8448-4b9b-4260-957a-802fedba2e83" (UID: "55ef8448-4b9b-4260-957a-802fedba2e83"). InnerVolumeSpecName "kube-api-access-hdzcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:04:25.249266 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.249243    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ef8448-4b9b-4260-957a-802fedba2e83-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55ef8448-4b9b-4260-957a-802fedba2e83" (UID: "55ef8448-4b9b-4260-957a-802fedba2e83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:04:25.259729 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.259702    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55ef8448-4b9b-4260-957a-802fedba2e83" (UID: "55ef8448-4b9b-4260-957a-802fedba2e83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:04:25.314593 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.314543    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55ef8448-4b9b-4260-957a-802fedba2e83" (UID: "55ef8448-4b9b-4260-957a-802fedba2e83"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:04:25.330122 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.330078    2569 generic.go:356] "Generic (PLEG): container finished" podID="55ef8448-4b9b-4260-957a-802fedba2e83" containerID="33a722d3e070efb89a52bfa148183d25997c16db184eb3a6fea6ab9001406c76" exitCode=0
Jul 30 00:04:25.330311 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.330122    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" event={"ID":"55ef8448-4b9b-4260-957a-802fedba2e83","Type":"ContainerDied","Data":"33a722d3e070efb89a52bfa148183d25997c16db184eb3a6fea6ab9001406c76"}
Jul 30 00:04:25.330311 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.330130    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"
Jul 30 00:04:25.330311 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.330229    2569 scope.go:117] "RemoveContainer" containerID="33a722d3e070efb89a52bfa148183d25997c16db184eb3a6fea6ab9001406c76"
Jul 30 00:04:25.330311 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.330217    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss" event={"ID":"55ef8448-4b9b-4260-957a-802fedba2e83","Type":"ContainerDied","Data":"d90423c53c40a32eb7c06f6196e1a7861ee4dbb4b9b9fa1661f13bd0e9a59667"}
Jul 30 00:04:25.344124 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.344068    2569 scope.go:117] "RemoveContainer" containerID="25e16d15577a0a51ba93fdfa6ba9acff5f87dd868f8ac790bc9afda5b52f11cf"
Jul 30 00:04:25.347833 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.347803    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:04:25.347946 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.347838    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-kserve-provision-location\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:04:25.347946 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.347856    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:04:25.347946 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.347870    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ef8448-4b9b-4260-957a-802fedba2e83-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:04:25.347946 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.347883    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ef8448-4b9b-4260-957a-802fedba2e83-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:04:25.347946 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.347896    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55ef8448-4b9b-4260-957a-802fedba2e83-cabundle-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:04:25.347946 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.347909    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdzcq\" (UniqueName: \"kubernetes.io/projected/55ef8448-4b9b-4260-957a-802fedba2e83-kube-api-access-hdzcq\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:04:25.352860 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.352817    2569 scope.go:117] "RemoveContainer" containerID="33a722d3e070efb89a52bfa148183d25997c16db184eb3a6fea6ab9001406c76"
Jul 30 00:04:25.353625 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:04:25.353332    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a722d3e070efb89a52bfa148183d25997c16db184eb3a6fea6ab9001406c76\": container with ID starting with 33a722d3e070efb89a52bfa148183d25997c16db184eb3a6fea6ab9001406c76 not found: ID does not exist" containerID="33a722d3e070efb89a52bfa148183d25997c16db184eb3a6fea6ab9001406c76"
Jul 30 00:04:25.353625 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.353373    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a722d3e070efb89a52bfa148183d25997c16db184eb3a6fea6ab9001406c76"} err="failed to get container status \"33a722d3e070efb89a52bfa148183d25997c16db184eb3a6fea6ab9001406c76\": rpc error: code = NotFound desc = could not find container \"33a722d3e070efb89a52bfa148183d25997c16db184eb3a6fea6ab9001406c76\": container with ID starting with 33a722d3e070efb89a52bfa148183d25997c16db184eb3a6fea6ab9001406c76 not found: ID does not exist"
Jul 30 00:04:25.353625 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.353398    2569 scope.go:117] "RemoveContainer" containerID="25e16d15577a0a51ba93fdfa6ba9acff5f87dd868f8ac790bc9afda5b52f11cf"
Jul 30 00:04:25.353899 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:04:25.353878    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e16d15577a0a51ba93fdfa6ba9acff5f87dd868f8ac790bc9afda5b52f11cf\": container with ID starting with 25e16d15577a0a51ba93fdfa6ba9acff5f87dd868f8ac790bc9afda5b52f11cf not found: ID does not exist" containerID="25e16d15577a0a51ba93fdfa6ba9acff5f87dd868f8ac790bc9afda5b52f11cf"
Jul 30 00:04:25.353961 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.353905    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e16d15577a0a51ba93fdfa6ba9acff5f87dd868f8ac790bc9afda5b52f11cf"} err="failed to get container status \"25e16d15577a0a51ba93fdfa6ba9acff5f87dd868f8ac790bc9afda5b52f11cf\": rpc error: code = NotFound desc = could not find container \"25e16d15577a0a51ba93fdfa6ba9acff5f87dd868f8ac790bc9afda5b52f11cf\": container with ID starting with 25e16d15577a0a51ba93fdfa6ba9acff5f87dd868f8ac790bc9afda5b52f11cf not found: ID does not exist"
Jul 30 00:04:25.355285 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.355256    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"]
Jul 30 00:04:25.359236 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.359208    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-689f8679cf-fm9ss"]
Jul 30 00:04:25.473982 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:25.473942    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55ef8448-4b9b-4260-957a-802fedba2e83" path="/var/lib/kubelet/pods/55ef8448-4b9b-4260-957a-802fedba2e83/volumes"
Jul 30 00:04:38.103592 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.103553    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"]
Jul 30 00:04:38.104057 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.103935    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ef8448-4b9b-4260-957a-802fedba2e83" containerName="storage-initializer"
Jul 30 00:04:38.104057 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.103955    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ef8448-4b9b-4260-957a-802fedba2e83" containerName="storage-initializer"
Jul 30 00:04:38.104057 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.103966    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ef8448-4b9b-4260-957a-802fedba2e83" containerName="main"
Jul 30 00:04:38.104057 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.103972    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ef8448-4b9b-4260-957a-802fedba2e83" containerName="main"
Jul 30 00:04:38.104057 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.104052    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="55ef8448-4b9b-4260-957a-802fedba2e83" containerName="main"
Jul 30 00:04:38.108391 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.108368    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.110456 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.110430    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Jul 30 00:04:38.110639 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.110444    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 30 00:04:38.110793 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.110473    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 30 00:04:38.110869 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.110517    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-4gdvp\""
Jul 30 00:04:38.110869 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.110553    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 30 00:04:38.118235 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.118209    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"]
Jul 30 00:04:38.258426 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.258385    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-tmp-dir\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.258661 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.258432    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nj59\" (UniqueName: \"kubernetes.io/projected/7a9275a2-a3f1-4b40-815c-b69bd63043a4-kube-api-access-5nj59\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.258661 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.258474    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-model-cache\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.258661 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.258531    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-home\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.258661 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.258599    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-dshm\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.258661 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.258623    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a9275a2-a3f1-4b40-815c-b69bd63043a4-tls-certs\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.258661 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.258651    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-kserve-provision-location\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.359602 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.359478    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-dshm\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.359602 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.359556    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a9275a2-a3f1-4b40-815c-b69bd63043a4-tls-certs\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.359602 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.359580    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-kserve-provision-location\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.359602 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.359612    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-tmp-dir\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.359972 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.359633    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nj59\" (UniqueName: \"kubernetes.io/projected/7a9275a2-a3f1-4b40-815c-b69bd63043a4-kube-api-access-5nj59\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.359972 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.359661    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-model-cache\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.359972 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.359703    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-home\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.360149 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.360025    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-kserve-provision-location\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.360149 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.360055    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-tmp-dir\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.360149 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.360075    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-home\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.360149 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.360125    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-model-cache\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.361730 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.361706    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-dshm\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.362009 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.361991    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a9275a2-a3f1-4b40-815c-b69bd63043a4-tls-certs\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.367421 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.367399    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nj59\" (UniqueName: \"kubernetes.io/projected/7a9275a2-a3f1-4b40-815c-b69bd63043a4-kube-api-access-5nj59\") pod \"tls-verification-test-kserve-5dcc8fd484-xr8xf\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.419797 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.419757    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:38.567371 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:38.567329    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"]
Jul 30 00:04:38.570376 ip-10-0-135-98 kubenswrapper[2569]: W0730 00:04:38.570351    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9275a2_a3f1_4b40_815c_b69bd63043a4.slice/crio-412f30f1af43c9ebed5931277c713c509076fffec54e37647270d231da31db98 WatchSource:0}: Error finding container 412f30f1af43c9ebed5931277c713c509076fffec54e37647270d231da31db98: Status 404 returned error can't find the container with id 412f30f1af43c9ebed5931277c713c509076fffec54e37647270d231da31db98
Jul 30 00:04:39.392610 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:39.392575    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" event={"ID":"7a9275a2-a3f1-4b40-815c-b69bd63043a4","Type":"ContainerStarted","Data":"192222ccc21efdab891dfe2f028be9485db51e8cbc6287c31eb6610c6e18cf98"}
Jul 30 00:04:39.393030 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:39.392636    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" event={"ID":"7a9275a2-a3f1-4b40-815c-b69bd63043a4","Type":"ContainerStarted","Data":"412f30f1af43c9ebed5931277c713c509076fffec54e37647270d231da31db98"}
Jul 30 00:04:44.414627 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:44.414589    2569 generic.go:356] "Generic (PLEG): container finished" podID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerID="192222ccc21efdab891dfe2f028be9485db51e8cbc6287c31eb6610c6e18cf98" exitCode=0
Jul 30 00:04:44.415037 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:44.414665    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" event={"ID":"7a9275a2-a3f1-4b40-815c-b69bd63043a4","Type":"ContainerDied","Data":"192222ccc21efdab891dfe2f028be9485db51e8cbc6287c31eb6610c6e18cf98"}
Jul 30 00:04:45.421172 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:45.421129    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" event={"ID":"7a9275a2-a3f1-4b40-815c-b69bd63043a4","Type":"ContainerStarted","Data":"3d1654c1cbf9d1359fffd34a8828d234122dea30bfe93b78c07351e679fba17b"}
Jul 30 00:04:45.444801 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:45.444727    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" podStartSLOduration=2.525785799 podStartE2EDuration="7.444706658s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.918920859s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:04:45.441844782 +0000 UTC m=+4692.447042205" watchObservedRunningTime="2026-07-30 00:04:45.444706658 +0000 UTC m=+4692.449904082"
Jul 30 00:04:46.420133 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:46.420094    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:48.419912 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:48.419870    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:04:48.421563 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:48.421529    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 30 00:04:58.420297 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:04:58.420252    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 30 00:05:08.420579 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:05:08.420515    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 30 00:05:18.421206 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:05:18.421156    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 30 00:05:28.421115 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:05:28.421058    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 30 00:05:38.421135 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:05:38.421090    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 30 00:05:48.420990 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:05:48.420938    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 30 00:05:58.420355 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:05:58.420257    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 30 00:06:08.420567 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:06:08.420516    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 30 00:06:18.420229 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:06:18.420187    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 30 00:06:28.420830 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:06:28.420781    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 30 00:06:33.829371 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:06:33.829248    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 30 00:06:33.837018 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:06:33.836995    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 30 00:06:38.430933 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:06:38.430894    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:06:38.439500 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:06:38.439455    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:06:50.591445 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:06:50.591410    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"]
Jul 30 00:07:05.614267 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:05.614223    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="main" containerID="cri-o://3d1654c1cbf9d1359fffd34a8828d234122dea30bfe93b78c07351e679fba17b" gracePeriod=45
Jul 30 00:07:06.363832 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.363807    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:07:06.496739 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.496707    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-model-cache\") pod \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") "
Jul 30 00:07:06.496908 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.496747    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-kserve-provision-location\") pod \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") "
Jul 30 00:07:06.496908 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.496810    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nj59\" (UniqueName: \"kubernetes.io/projected/7a9275a2-a3f1-4b40-815c-b69bd63043a4-kube-api-access-5nj59\") pod \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") "
Jul 30 00:07:06.496908 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.496838    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-dshm\") pod \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") "
Jul 30 00:07:06.496908 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.496886    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-home\") pod \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") "
Jul 30 00:07:06.497102 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.496919    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a9275a2-a3f1-4b40-815c-b69bd63043a4-tls-certs\") pod \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") "
Jul 30 00:07:06.497102 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.496948    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-tmp-dir\") pod \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\" (UID: \"7a9275a2-a3f1-4b40-815c-b69bd63043a4\") "
Jul 30 00:07:06.497102 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.497062    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a9275a2-a3f1-4b40-815c-b69bd63043a4" (UID: "7a9275a2-a3f1-4b40-815c-b69bd63043a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:07:06.497425 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.497280    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:07:06.497660 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.497635    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-home" (OuterVolumeSpecName: "home") pod "7a9275a2-a3f1-4b40-815c-b69bd63043a4" (UID: "7a9275a2-a3f1-4b40-815c-b69bd63043a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:07:06.499194 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.499168    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-dshm" (OuterVolumeSpecName: "dshm") pod "7a9275a2-a3f1-4b40-815c-b69bd63043a4" (UID: "7a9275a2-a3f1-4b40-815c-b69bd63043a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:07:06.499569 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.499541    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a9275a2-a3f1-4b40-815c-b69bd63043a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a9275a2-a3f1-4b40-815c-b69bd63043a4" (UID: "7a9275a2-a3f1-4b40-815c-b69bd63043a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:07:06.499820 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.499783    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9275a2-a3f1-4b40-815c-b69bd63043a4-kube-api-access-5nj59" (OuterVolumeSpecName: "kube-api-access-5nj59") pod "7a9275a2-a3f1-4b40-815c-b69bd63043a4" (UID: "7a9275a2-a3f1-4b40-815c-b69bd63043a4"). InnerVolumeSpecName "kube-api-access-5nj59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:07:06.516621 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.516580    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a9275a2-a3f1-4b40-815c-b69bd63043a4" (UID: "7a9275a2-a3f1-4b40-815c-b69bd63043a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:07:06.566205 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.566150    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a9275a2-a3f1-4b40-815c-b69bd63043a4" (UID: "7a9275a2-a3f1-4b40-815c-b69bd63043a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:07:06.598491 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.598455    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:07:06.598609 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.598510    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a9275a2-a3f1-4b40-815c-b69bd63043a4-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:07:06.598609 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.598524    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:07:06.598609 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.598533    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-kserve-provision-location\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:07:06.598609 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.598543    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nj59\" (UniqueName: \"kubernetes.io/projected/7a9275a2-a3f1-4b40-815c-b69bd63043a4-kube-api-access-5nj59\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:07:06.598609 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:06.598551    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a9275a2-a3f1-4b40-815c-b69bd63043a4-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:07:07.047356 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:07.047322    2569 generic.go:356] "Generic (PLEG): container finished" podID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerID="3d1654c1cbf9d1359fffd34a8828d234122dea30bfe93b78c07351e679fba17b" exitCode=0
Jul 30 00:07:07.047356 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:07.047361    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" event={"ID":"7a9275a2-a3f1-4b40-815c-b69bd63043a4","Type":"ContainerDied","Data":"3d1654c1cbf9d1359fffd34a8828d234122dea30bfe93b78c07351e679fba17b"}
Jul 30 00:07:07.047940 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:07.047391    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"
Jul 30 00:07:07.047940 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:07.047436    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf" event={"ID":"7a9275a2-a3f1-4b40-815c-b69bd63043a4","Type":"ContainerDied","Data":"412f30f1af43c9ebed5931277c713c509076fffec54e37647270d231da31db98"}
Jul 30 00:07:07.047940 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:07.047459    2569 scope.go:117] "RemoveContainer" containerID="3d1654c1cbf9d1359fffd34a8828d234122dea30bfe93b78c07351e679fba17b"
Jul 30 00:07:07.056571 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:07.056508    2569 scope.go:117] "RemoveContainer" containerID="192222ccc21efdab891dfe2f028be9485db51e8cbc6287c31eb6610c6e18cf98"
Jul 30 00:07:07.064249 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:07.064227    2569 scope.go:117] "RemoveContainer" containerID="3d1654c1cbf9d1359fffd34a8828d234122dea30bfe93b78c07351e679fba17b"
Jul 30 00:07:07.064536 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:07:07.064466    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1654c1cbf9d1359fffd34a8828d234122dea30bfe93b78c07351e679fba17b\": container with ID starting with 3d1654c1cbf9d1359fffd34a8828d234122dea30bfe93b78c07351e679fba17b not found: ID does not exist" containerID="3d1654c1cbf9d1359fffd34a8828d234122dea30bfe93b78c07351e679fba17b"
Jul 30 00:07:07.064536 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:07.064512    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1654c1cbf9d1359fffd34a8828d234122dea30bfe93b78c07351e679fba17b"} err="failed to get container status \"3d1654c1cbf9d1359fffd34a8828d234122dea30bfe93b78c07351e679fba17b\": rpc error: code = NotFound desc = could not find container \"3d1654c1cbf9d1359fffd34a8828d234122dea30bfe93b78c07351e679fba17b\": container with ID starting with 3d1654c1cbf9d1359fffd34a8828d234122dea30bfe93b78c07351e679fba17b not found: ID does not exist"
Jul 30 00:07:07.064536 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:07.064535    2569 scope.go:117] "RemoveContainer" containerID="192222ccc21efdab891dfe2f028be9485db51e8cbc6287c31eb6610c6e18cf98"
Jul 30 00:07:07.064772 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:07:07.064755    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192222ccc21efdab891dfe2f028be9485db51e8cbc6287c31eb6610c6e18cf98\": container with ID starting with 192222ccc21efdab891dfe2f028be9485db51e8cbc6287c31eb6610c6e18cf98 not found: ID does not exist" containerID="192222ccc21efdab891dfe2f028be9485db51e8cbc6287c31eb6610c6e18cf98"
Jul 30 00:07:07.064816 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:07.064780    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192222ccc21efdab891dfe2f028be9485db51e8cbc6287c31eb6610c6e18cf98"} err="failed to get container status \"192222ccc21efdab891dfe2f028be9485db51e8cbc6287c31eb6610c6e18cf98\": rpc error: code = NotFound desc = could not find container \"192222ccc21efdab891dfe2f028be9485db51e8cbc6287c31eb6610c6e18cf98\": container with ID starting with 192222ccc21efdab891dfe2f028be9485db51e8cbc6287c31eb6610c6e18cf98 not found: ID does not exist"
Jul 30 00:07:07.068089 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:07.068068    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"]
Jul 30 00:07:07.071326 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:07.071305    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5dcc8fd484-xr8xf"]
Jul 30 00:07:07.474856 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:07.474824    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" path="/var/lib/kubelet/pods/7a9275a2-a3f1-4b40-815c-b69bd63043a4/volumes"
Jul 30 00:07:18.300156 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.300102    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"]
Jul 30 00:07:18.300682 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.300457    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="main"
Jul 30 00:07:18.300682 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.300472    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="main"
Jul 30 00:07:18.300682 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.300508    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="storage-initializer"
Jul 30 00:07:18.300682 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.300516    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="storage-initializer"
Jul 30 00:07:18.300682 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.300573    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a9275a2-a3f1-4b40-815c-b69bd63043a4" containerName="main"
Jul 30 00:07:18.305309 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.305288    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.307272 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.307245    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Jul 30 00:07:18.307272 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.307259    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 30 00:07:18.307540 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.307246    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-cpsfq\""
Jul 30 00:07:18.307540 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.307286    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 30 00:07:18.307540 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.307451    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 30 00:07:18.313619 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.313598    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"]
Jul 30 00:07:18.392774 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.392730    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-tmp-dir\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.392774 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.392774    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-home\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.393021 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.392803    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-dshm\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.393021 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.392871    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a53154-3cdf-40d5-a493-35ba1c757714-tls-certs\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.393021 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.392909    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-model-cache\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.393021 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.392926    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tcq8\" (UniqueName: \"kubernetes.io/projected/f2a53154-3cdf-40d5-a493-35ba1c757714-kube-api-access-6tcq8\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.393021 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.392962    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-kserve-provision-location\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.493545 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.493492    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-kserve-provision-location\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.493734 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.493607    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-tmp-dir\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.493734 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.493643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-home\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.493997 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.493960    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-kserve-provision-location\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.494146 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.494010    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-home\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.494146 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.493736    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-dshm\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.494146 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.494117    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a53154-3cdf-40d5-a493-35ba1c757714-tls-certs\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.494326 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.494165    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-model-cache\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.494326 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.494191    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tcq8\" (UniqueName: \"kubernetes.io/projected/f2a53154-3cdf-40d5-a493-35ba1c757714-kube-api-access-6tcq8\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.494326 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.494294    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-tmp-dir\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.494543 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.494517    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-model-cache\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.496982 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.496935    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-dshm\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.497193 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.497169    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a53154-3cdf-40d5-a493-35ba1c757714-tls-certs\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.508962 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.508934    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tcq8\" (UniqueName: \"kubernetes.io/projected/f2a53154-3cdf-40d5-a493-35ba1c757714-kube-api-access-6tcq8\") pod \"prestop-hook-test-kserve-6597646664-g55h6\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.617860 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.617773    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:18.754923 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:18.754895    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"]
Jul 30 00:07:18.757436 ip-10-0-135-98 kubenswrapper[2569]: W0730 00:07:18.757410    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a53154_3cdf_40d5_a493_35ba1c757714.slice/crio-a8c85c71da84a30d3e1bfe857f7cb01c5f99dfd650629a2e85860e9557fbf952 WatchSource:0}: Error finding container a8c85c71da84a30d3e1bfe857f7cb01c5f99dfd650629a2e85860e9557fbf952: Status 404 returned error can't find the container with id a8c85c71da84a30d3e1bfe857f7cb01c5f99dfd650629a2e85860e9557fbf952
Jul 30 00:07:19.099916 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:19.099876    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" event={"ID":"f2a53154-3cdf-40d5-a493-35ba1c757714","Type":"ContainerStarted","Data":"db14b2d5caef0ac9db18c9bca8f528872228593595e73a8e62b979d23a89a3e4"}
Jul 30 00:07:19.100099 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:19.099939    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" event={"ID":"f2a53154-3cdf-40d5-a493-35ba1c757714","Type":"ContainerStarted","Data":"a8c85c71da84a30d3e1bfe857f7cb01c5f99dfd650629a2e85860e9557fbf952"}
Jul 30 00:07:27.135036 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:27.134987    2569 generic.go:356] "Generic (PLEG): container finished" podID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerID="db14b2d5caef0ac9db18c9bca8f528872228593595e73a8e62b979d23a89a3e4" exitCode=0
Jul 30 00:07:27.135400 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:27.135070    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" event={"ID":"f2a53154-3cdf-40d5-a493-35ba1c757714","Type":"ContainerDied","Data":"db14b2d5caef0ac9db18c9bca8f528872228593595e73a8e62b979d23a89a3e4"}
Jul 30 00:07:28.141571 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:28.141532    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" event={"ID":"f2a53154-3cdf-40d5-a493-35ba1c757714","Type":"ContainerStarted","Data":"8356c38f5fd67fc170140930b284bc754e4de8e4c554a64f2bd7775ec0fb0719"}
Jul 30 00:07:28.161878 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:28.161818    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" podStartSLOduration=1.942176862 podStartE2EDuration="10.16180088s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.219624018s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:07:28.160422384 +0000 UTC m=+4855.165619820" watchObservedRunningTime="2026-07-30 00:07:28.16180088 +0000 UTC m=+4855.166998303"
Jul 30 00:07:28.618523 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:28.618471    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:28.618792 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:28.618558    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:07:28.619970 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:28.619934    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 30 00:07:38.619151 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:38.619102    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 30 00:07:48.619217 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:48.619169    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 30 00:07:58.618840 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:07:58.618785    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 30 00:08:08.618719 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:08:08.618668    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 30 00:08:18.618921 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:08:18.618860    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 30 00:08:28.618385 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:08:28.618343    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 30 00:08:38.618570 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:08:38.618517    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 30 00:08:48.619078 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:08:48.619019    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 30 00:08:58.619157 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:08:58.619061    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 30 00:09:08.619239 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:08.619184    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 30 00:09:18.618499 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:18.618426    2569 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 30 00:09:28.628896 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:28.628853    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:09:28.642467 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:28.642417    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:09:32.556689 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:32.556655    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"]
Jul 30 00:09:47.579175 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:47.579131    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main" containerID="cri-o://8356c38f5fd67fc170140930b284bc754e4de8e4c554a64f2bd7775ec0fb0719" gracePeriod=45
Jul 30 00:09:48.337366 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.337339    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:09:48.419540 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.419439    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-kserve-provision-location\") pod \"f2a53154-3cdf-40d5-a493-35ba1c757714\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") "
Jul 30 00:09:48.419540 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.419474    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-dshm\") pod \"f2a53154-3cdf-40d5-a493-35ba1c757714\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") "
Jul 30 00:09:48.419540 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.419522    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-model-cache\") pod \"f2a53154-3cdf-40d5-a493-35ba1c757714\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") "
Jul 30 00:09:48.419850 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.419562    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-tmp-dir\") pod \"f2a53154-3cdf-40d5-a493-35ba1c757714\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") "
Jul 30 00:09:48.419850 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.419602    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a53154-3cdf-40d5-a493-35ba1c757714-tls-certs\") pod \"f2a53154-3cdf-40d5-a493-35ba1c757714\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") "
Jul 30 00:09:48.419850 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.419626    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-home\") pod \"f2a53154-3cdf-40d5-a493-35ba1c757714\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") "
Jul 30 00:09:48.419850 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.419654    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tcq8\" (UniqueName: \"kubernetes.io/projected/f2a53154-3cdf-40d5-a493-35ba1c757714-kube-api-access-6tcq8\") pod \"f2a53154-3cdf-40d5-a493-35ba1c757714\" (UID: \"f2a53154-3cdf-40d5-a493-35ba1c757714\") "
Jul 30 00:09:48.419850 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.419829    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2a53154-3cdf-40d5-a493-35ba1c757714" (UID: "f2a53154-3cdf-40d5-a493-35ba1c757714"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:09:48.420444 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.420405    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-home" (OuterVolumeSpecName: "home") pod "f2a53154-3cdf-40d5-a493-35ba1c757714" (UID: "f2a53154-3cdf-40d5-a493-35ba1c757714"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:09:48.421954 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.421923    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2a53154-3cdf-40d5-a493-35ba1c757714-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2a53154-3cdf-40d5-a493-35ba1c757714" (UID: "f2a53154-3cdf-40d5-a493-35ba1c757714"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:09:48.422226 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.422201    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a53154-3cdf-40d5-a493-35ba1c757714-kube-api-access-6tcq8" (OuterVolumeSpecName: "kube-api-access-6tcq8") pod "f2a53154-3cdf-40d5-a493-35ba1c757714" (UID: "f2a53154-3cdf-40d5-a493-35ba1c757714"). InnerVolumeSpecName "kube-api-access-6tcq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:09:48.422226 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.422213    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-dshm" (OuterVolumeSpecName: "dshm") pod "f2a53154-3cdf-40d5-a493-35ba1c757714" (UID: "f2a53154-3cdf-40d5-a493-35ba1c757714"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:09:48.437537 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.437470    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2a53154-3cdf-40d5-a493-35ba1c757714" (UID: "f2a53154-3cdf-40d5-a493-35ba1c757714"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:09:48.487303 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.487260    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2a53154-3cdf-40d5-a493-35ba1c757714" (UID: "f2a53154-3cdf-40d5-a493-35ba1c757714"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:09:48.520804 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.520768    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2a53154-3cdf-40d5-a493-35ba1c757714-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:09:48.520804 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.520803    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:09:48.521058 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.520817    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tcq8\" (UniqueName: \"kubernetes.io/projected/f2a53154-3cdf-40d5-a493-35ba1c757714-kube-api-access-6tcq8\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:09:48.521058 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.520831    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-kserve-provision-location\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:09:48.521058 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.520846    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:09:48.521058 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.520859    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:09:48.521058 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.520871    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2a53154-3cdf-40d5-a493-35ba1c757714-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:09:48.739061 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.739031    2569 generic.go:356] "Generic (PLEG): container finished" podID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerID="8356c38f5fd67fc170140930b284bc754e4de8e4c554a64f2bd7775ec0fb0719" exitCode=0
Jul 30 00:09:48.739498 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.739081    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"
Jul 30 00:09:48.739498 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.739121    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" event={"ID":"f2a53154-3cdf-40d5-a493-35ba1c757714","Type":"ContainerDied","Data":"8356c38f5fd67fc170140930b284bc754e4de8e4c554a64f2bd7775ec0fb0719"}
Jul 30 00:09:48.739498 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.739184    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6" event={"ID":"f2a53154-3cdf-40d5-a493-35ba1c757714","Type":"ContainerDied","Data":"a8c85c71da84a30d3e1bfe857f7cb01c5f99dfd650629a2e85860e9557fbf952"}
Jul 30 00:09:48.739498 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.739209    2569 scope.go:117] "RemoveContainer" containerID="8356c38f5fd67fc170140930b284bc754e4de8e4c554a64f2bd7775ec0fb0719"
Jul 30 00:09:48.748257 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.748235    2569 scope.go:117] "RemoveContainer" containerID="db14b2d5caef0ac9db18c9bca8f528872228593595e73a8e62b979d23a89a3e4"
Jul 30 00:09:48.755956 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.755932    2569 scope.go:117] "RemoveContainer" containerID="8356c38f5fd67fc170140930b284bc754e4de8e4c554a64f2bd7775ec0fb0719"
Jul 30 00:09:48.756256 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:09:48.756232    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8356c38f5fd67fc170140930b284bc754e4de8e4c554a64f2bd7775ec0fb0719\": container with ID starting with 8356c38f5fd67fc170140930b284bc754e4de8e4c554a64f2bd7775ec0fb0719 not found: ID does not exist" containerID="8356c38f5fd67fc170140930b284bc754e4de8e4c554a64f2bd7775ec0fb0719"
Jul 30 00:09:48.756317 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.756270    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8356c38f5fd67fc170140930b284bc754e4de8e4c554a64f2bd7775ec0fb0719"} err="failed to get container status \"8356c38f5fd67fc170140930b284bc754e4de8e4c554a64f2bd7775ec0fb0719\": rpc error: code = NotFound desc = could not find container \"8356c38f5fd67fc170140930b284bc754e4de8e4c554a64f2bd7775ec0fb0719\": container with ID starting with 8356c38f5fd67fc170140930b284bc754e4de8e4c554a64f2bd7775ec0fb0719 not found: ID does not exist"
Jul 30 00:09:48.756317 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.756295    2569 scope.go:117] "RemoveContainer" containerID="db14b2d5caef0ac9db18c9bca8f528872228593595e73a8e62b979d23a89a3e4"
Jul 30 00:09:48.756578 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:09:48.756556    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db14b2d5caef0ac9db18c9bca8f528872228593595e73a8e62b979d23a89a3e4\": container with ID starting with db14b2d5caef0ac9db18c9bca8f528872228593595e73a8e62b979d23a89a3e4 not found: ID does not exist" containerID="db14b2d5caef0ac9db18c9bca8f528872228593595e73a8e62b979d23a89a3e4"
Jul 30 00:09:48.756686 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.756584    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db14b2d5caef0ac9db18c9bca8f528872228593595e73a8e62b979d23a89a3e4"} err="failed to get container status \"db14b2d5caef0ac9db18c9bca8f528872228593595e73a8e62b979d23a89a3e4\": rpc error: code = NotFound desc = could not find container \"db14b2d5caef0ac9db18c9bca8f528872228593595e73a8e62b979d23a89a3e4\": container with ID starting with db14b2d5caef0ac9db18c9bca8f528872228593595e73a8e62b979d23a89a3e4 not found: ID does not exist"
Jul 30 00:09:48.760532 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.760508    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"]
Jul 30 00:09:48.764397 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:48.764374    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-g55h6"]
Jul 30 00:09:49.473470 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:09:49.473430    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" path="/var/lib/kubelet/pods/f2a53154-3cdf-40d5-a493-35ba1c757714/volumes"
Jul 30 00:10:47.747925 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.747886    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"]
Jul 30 00:10:47.748350 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.748196    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="storage-initializer"
Jul 30 00:10:47.748350 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.748207    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="storage-initializer"
Jul 30 00:10:47.748350 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.748236    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main"
Jul 30 00:10:47.748350 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.748243    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main"
Jul 30 00:10:47.748350 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.748294    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a53154-3cdf-40d5-a493-35ba1c757714" containerName="main"
Jul 30 00:10:47.752387 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.752366    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.754922 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.754897    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-2cct7\""
Jul 30 00:10:47.755036 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.754929    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 30 00:10:47.755036 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.754966    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 30 00:10:47.755036 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.754992    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 30 00:10:47.761947 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.761921    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"]
Jul 30 00:10:47.834767 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.834727    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/749bfda2-72b1-4200-8109-d6e2c25073ea-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.834943 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.834781    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.834943 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.834843    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.834943 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.834908    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvb9h\" (UniqueName: \"kubernetes.io/projected/749bfda2-72b1-4200-8109-d6e2c25073ea-kube-api-access-zvb9h\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.835074 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.834941    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.835074 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.834971    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.935391 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.935354    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.935603 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.935399    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvb9h\" (UniqueName: \"kubernetes.io/projected/749bfda2-72b1-4200-8109-d6e2c25073ea-kube-api-access-zvb9h\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.935603 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.935428    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.935603 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.935551    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.935726 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.935613    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/749bfda2-72b1-4200-8109-d6e2c25073ea-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.935726 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.935660    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.935842 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.935818    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.935903 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.935867    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.936004 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.935983    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.937909 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.937887    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.938160 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.938144    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/749bfda2-72b1-4200-8109-d6e2c25073ea-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:47.942376 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:47.942357    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvb9h\" (UniqueName: \"kubernetes.io/projected/749bfda2-72b1-4200-8109-d6e2c25073ea-kube-api-access-zvb9h\") pod \"rolling-upgrade-test-kserve-74b79d7f77-szr4q\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:48.064393 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:48.064303    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:48.196142 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:48.196114    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"]
Jul 30 00:10:48.198857 ip-10-0-135-98 kubenswrapper[2569]: W0730 00:10:48.198826    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749bfda2_72b1_4200_8109_d6e2c25073ea.slice/crio-8863cf850c22baa4362953c080fc92327129675baca25353c566fb1485021dce WatchSource:0}: Error finding container 8863cf850c22baa4362953c080fc92327129675baca25353c566fb1485021dce: Status 404 returned error can't find the container with id 8863cf850c22baa4362953c080fc92327129675baca25353c566fb1485021dce
Jul 30 00:10:48.978259 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:48.978215    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q" event={"ID":"749bfda2-72b1-4200-8109-d6e2c25073ea","Type":"ContainerStarted","Data":"e97ad91d9e362ec93bbb6f82102e45e386ee8321f47346ecb7475b0f1fb48190"}
Jul 30 00:10:48.978694 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:48.978277    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q" event={"ID":"749bfda2-72b1-4200-8109-d6e2c25073ea","Type":"ContainerStarted","Data":"8863cf850c22baa4362953c080fc92327129675baca25353c566fb1485021dce"}
Jul 30 00:10:48.998752 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:48.998694    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q" podStartSLOduration=1.9986796820000001 podStartE2EDuration="1.998679682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:10:48.996904543 +0000 UTC m=+5056.002101966" watchObservedRunningTime="2026-07-30 00:10:48.998679682 +0000 UTC m=+5056.003877104"
Jul 30 00:10:49.065078 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:49.065016    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:58.065155 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:58.065116    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:58.077309 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:58.077282    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:10:59.035749 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:10:59.035715    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:11:33.855767 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:33.855649    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 30 00:11:33.867722 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:33.867700    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 30 00:11:54.175025 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:54.174988    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"]
Jul 30 00:11:54.183881 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:11:54.183844    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 30 00:11:54.183881 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 30 00:11:54.183881 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 30 00:11:54.183881 ip-10-0-135-98 kubenswrapper[2569]:  > containerID="e97ad91d9e362ec93bbb6f82102e45e386ee8321f47346ecb7475b0f1fb48190" cmd=["/bin/sleep","15"]
Jul 30 00:11:54.184123 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:54.183898    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q" message="" err=<
Jul 30 00:11:54.184123 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 30 00:11:54.184123 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 30 00:11:54.184123 ip-10-0-135-98 kubenswrapper[2569]:  >
Jul 30 00:11:54.184123 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:11:54.183910    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 30 00:11:54.184123 ip-10-0-135-98 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 30 00:11:54.184123 ip-10-0-135-98 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 30 00:11:54.184123 ip-10-0-135-98 kubenswrapper[2569]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q" podUID="749bfda2-72b1-4200-8109-d6e2c25073ea" containerName="main" containerID="cri-o://e97ad91d9e362ec93bbb6f82102e45e386ee8321f47346ecb7475b0f1fb48190"
Jul 30 00:11:54.184123 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:54.183943    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q" podUID="749bfda2-72b1-4200-8109-d6e2c25073ea" containerName="main" containerID="cri-o://e97ad91d9e362ec93bbb6f82102e45e386ee8321f47346ecb7475b0f1fb48190" gracePeriod=60
Jul 30 00:11:54.934209 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:54.934183    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:11:55.008845 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.008814    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-home\") pod \"749bfda2-72b1-4200-8109-d6e2c25073ea\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") "
Jul 30 00:11:55.009038 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.008864    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-model-cache\") pod \"749bfda2-72b1-4200-8109-d6e2c25073ea\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") "
Jul 30 00:11:55.009038 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.008884    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-tmp-dir\") pod \"749bfda2-72b1-4200-8109-d6e2c25073ea\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") "
Jul 30 00:11:55.009038 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.008951    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvb9h\" (UniqueName: \"kubernetes.io/projected/749bfda2-72b1-4200-8109-d6e2c25073ea-kube-api-access-zvb9h\") pod \"749bfda2-72b1-4200-8109-d6e2c25073ea\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") "
Jul 30 00:11:55.009038 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.008977    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-dshm\") pod \"749bfda2-72b1-4200-8109-d6e2c25073ea\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") "
Jul 30 00:11:55.009038 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.009007    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/749bfda2-72b1-4200-8109-d6e2c25073ea-tls-certs\") pod \"749bfda2-72b1-4200-8109-d6e2c25073ea\" (UID: \"749bfda2-72b1-4200-8109-d6e2c25073ea\") "
Jul 30 00:11:55.009286 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.009102    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-home" (OuterVolumeSpecName: "home") pod "749bfda2-72b1-4200-8109-d6e2c25073ea" (UID: "749bfda2-72b1-4200-8109-d6e2c25073ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:11:55.009286 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.009127    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "749bfda2-72b1-4200-8109-d6e2c25073ea" (UID: "749bfda2-72b1-4200-8109-d6e2c25073ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:11:55.009286 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.009219    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "749bfda2-72b1-4200-8109-d6e2c25073ea" (UID: "749bfda2-72b1-4200-8109-d6e2c25073ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:11:55.009423 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.009295    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:11:55.009423 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.009317    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:11:55.009423 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.009331    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:11:55.011202 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.011179    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-dshm" (OuterVolumeSpecName: "dshm") pod "749bfda2-72b1-4200-8109-d6e2c25073ea" (UID: "749bfda2-72b1-4200-8109-d6e2c25073ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:11:55.011312 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.011275    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/749bfda2-72b1-4200-8109-d6e2c25073ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "749bfda2-72b1-4200-8109-d6e2c25073ea" (UID: "749bfda2-72b1-4200-8109-d6e2c25073ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:11:55.011358 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.011341    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/749bfda2-72b1-4200-8109-d6e2c25073ea-kube-api-access-zvb9h" (OuterVolumeSpecName: "kube-api-access-zvb9h") pod "749bfda2-72b1-4200-8109-d6e2c25073ea" (UID: "749bfda2-72b1-4200-8109-d6e2c25073ea"). InnerVolumeSpecName "kube-api-access-zvb9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:11:55.110711 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.110626    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvb9h\" (UniqueName: \"kubernetes.io/projected/749bfda2-72b1-4200-8109-d6e2c25073ea-kube-api-access-zvb9h\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:11:55.110711 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.110657    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/749bfda2-72b1-4200-8109-d6e2c25073ea-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:11:55.110711 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.110666    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/749bfda2-72b1-4200-8109-d6e2c25073ea-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:11:55.246065 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.245975    2569 generic.go:356] "Generic (PLEG): container finished" podID="749bfda2-72b1-4200-8109-d6e2c25073ea" containerID="e97ad91d9e362ec93bbb6f82102e45e386ee8321f47346ecb7475b0f1fb48190" exitCode=0
Jul 30 00:11:55.246065 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.246018    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"
Jul 30 00:11:55.246065 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.246052    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q" event={"ID":"749bfda2-72b1-4200-8109-d6e2c25073ea","Type":"ContainerDied","Data":"e97ad91d9e362ec93bbb6f82102e45e386ee8321f47346ecb7475b0f1fb48190"}
Jul 30 00:11:55.246609 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.246104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q" event={"ID":"749bfda2-72b1-4200-8109-d6e2c25073ea","Type":"ContainerDied","Data":"8863cf850c22baa4362953c080fc92327129675baca25353c566fb1485021dce"}
Jul 30 00:11:55.246609 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.246128    2569 scope.go:117] "RemoveContainer" containerID="e97ad91d9e362ec93bbb6f82102e45e386ee8321f47346ecb7475b0f1fb48190"
Jul 30 00:11:55.255073 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.255051    2569 scope.go:117] "RemoveContainer" containerID="e97ad91d9e362ec93bbb6f82102e45e386ee8321f47346ecb7475b0f1fb48190"
Jul 30 00:11:55.255401 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:11:55.255385    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97ad91d9e362ec93bbb6f82102e45e386ee8321f47346ecb7475b0f1fb48190\": container with ID starting with e97ad91d9e362ec93bbb6f82102e45e386ee8321f47346ecb7475b0f1fb48190 not found: ID does not exist" containerID="e97ad91d9e362ec93bbb6f82102e45e386ee8321f47346ecb7475b0f1fb48190"
Jul 30 00:11:55.255453 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.255408    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97ad91d9e362ec93bbb6f82102e45e386ee8321f47346ecb7475b0f1fb48190"} err="failed to get container status \"e97ad91d9e362ec93bbb6f82102e45e386ee8321f47346ecb7475b0f1fb48190\": rpc error: code = NotFound desc = could not find container \"e97ad91d9e362ec93bbb6f82102e45e386ee8321f47346ecb7475b0f1fb48190\": container with ID starting with e97ad91d9e362ec93bbb6f82102e45e386ee8321f47346ecb7475b0f1fb48190 not found: ID does not exist"
Jul 30 00:11:55.266580 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.266559    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"]
Jul 30 00:11:55.271194 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.271173    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-szr4q"]
Jul 30 00:11:55.474453 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:11:55.474418    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="749bfda2-72b1-4200-8109-d6e2c25073ea" path="/var/lib/kubelet/pods/749bfda2-72b1-4200-8109-d6e2c25073ea/volumes"
Jul 30 00:12:16.226792 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:16.226751    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-7586c4769-btsn4"]
Jul 30 00:12:16.227330 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:16.227093    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="749bfda2-72b1-4200-8109-d6e2c25073ea" containerName="main"
Jul 30 00:12:16.227330 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:16.227109    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="749bfda2-72b1-4200-8109-d6e2c25073ea" containerName="main"
Jul 30 00:12:16.227330 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:16.227161    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="749bfda2-72b1-4200-8109-d6e2c25073ea" containerName="main"
Jul 30 00:12:16.230170 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:16.230150    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7586c4769-btsn4"
Jul 30 00:12:16.237432 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:16.237335    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7586c4769-btsn4"]
Jul 30 00:12:16.397981 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:16.397932    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f28c7d91-fe2f-4c45-a927-22fb7b3aac6b-cert\") pod \"llmisvc-controller-manager-7586c4769-btsn4\" (UID: \"f28c7d91-fe2f-4c45-a927-22fb7b3aac6b\") " pod="kserve/llmisvc-controller-manager-7586c4769-btsn4"
Jul 30 00:12:16.398171 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:16.397998    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t48rc\" (UniqueName: \"kubernetes.io/projected/f28c7d91-fe2f-4c45-a927-22fb7b3aac6b-kube-api-access-t48rc\") pod \"llmisvc-controller-manager-7586c4769-btsn4\" (UID: \"f28c7d91-fe2f-4c45-a927-22fb7b3aac6b\") " pod="kserve/llmisvc-controller-manager-7586c4769-btsn4"
Jul 30 00:12:16.498630 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:16.498536    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f28c7d91-fe2f-4c45-a927-22fb7b3aac6b-cert\") pod \"llmisvc-controller-manager-7586c4769-btsn4\" (UID: \"f28c7d91-fe2f-4c45-a927-22fb7b3aac6b\") " pod="kserve/llmisvc-controller-manager-7586c4769-btsn4"
Jul 30 00:12:16.498630 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:16.498590    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t48rc\" (UniqueName: \"kubernetes.io/projected/f28c7d91-fe2f-4c45-a927-22fb7b3aac6b-kube-api-access-t48rc\") pod \"llmisvc-controller-manager-7586c4769-btsn4\" (UID: \"f28c7d91-fe2f-4c45-a927-22fb7b3aac6b\") " pod="kserve/llmisvc-controller-manager-7586c4769-btsn4"
Jul 30 00:12:16.501022 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:16.500996    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f28c7d91-fe2f-4c45-a927-22fb7b3aac6b-cert\") pod \"llmisvc-controller-manager-7586c4769-btsn4\" (UID: \"f28c7d91-fe2f-4c45-a927-22fb7b3aac6b\") " pod="kserve/llmisvc-controller-manager-7586c4769-btsn4"
Jul 30 00:12:16.505809 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:16.505783    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t48rc\" (UniqueName: \"kubernetes.io/projected/f28c7d91-fe2f-4c45-a927-22fb7b3aac6b-kube-api-access-t48rc\") pod \"llmisvc-controller-manager-7586c4769-btsn4\" (UID: \"f28c7d91-fe2f-4c45-a927-22fb7b3aac6b\") " pod="kserve/llmisvc-controller-manager-7586c4769-btsn4"
Jul 30 00:12:16.541737 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:16.541705    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7586c4769-btsn4"
Jul 30 00:12:16.674264 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:16.674225    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7586c4769-btsn4"]
Jul 30 00:12:16.677236 ip-10-0-135-98 kubenswrapper[2569]: W0730 00:12:16.677202    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf28c7d91_fe2f_4c45_a927_22fb7b3aac6b.slice/crio-1b36a94081cf37ecb0137519ffc2100ca4444f755c5215d9f07b69a889ddfb7d WatchSource:0}: Error finding container 1b36a94081cf37ecb0137519ffc2100ca4444f755c5215d9f07b69a889ddfb7d: Status 404 returned error can't find the container with id 1b36a94081cf37ecb0137519ffc2100ca4444f755c5215d9f07b69a889ddfb7d
Jul 30 00:12:16.678752 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:16.678729    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 00:12:17.338380 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:17.338347    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7586c4769-btsn4" event={"ID":"f28c7d91-fe2f-4c45-a927-22fb7b3aac6b","Type":"ContainerStarted","Data":"59e5a361873b450c2e286ca856929d62136a18646e9acecc934a8eb5a53ac423"}
Jul 30 00:12:17.338793 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:17.338407    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7586c4769-btsn4" event={"ID":"f28c7d91-fe2f-4c45-a927-22fb7b3aac6b","Type":"ContainerStarted","Data":"1b36a94081cf37ecb0137519ffc2100ca4444f755c5215d9f07b69a889ddfb7d"}
Jul 30 00:12:17.338793 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:17.338475    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-7586c4769-btsn4"
Jul 30 00:12:17.356097 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:17.356048    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-7586c4769-btsn4" podStartSLOduration=0.910589493 podStartE2EDuration="1.356032039s" totalImagesPullingTime="445.442546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:12:17.354406609 +0000 UTC m=+5144.359604032" watchObservedRunningTime="2026-07-30 00:12:17.356032039 +0000 UTC m=+5144.361229463"
Jul 30 00:12:48.344987 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:48.344952    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-7586c4769-btsn4"
Jul 30 00:12:48.385396 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:48.385358    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"]
Jul 30 00:12:48.385758 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:48.385724    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l" podUID="f759b397-beac-4707-ba7f-1c1bb88128c1" containerName="manager" containerID="cri-o://0f46d27ecbfd6ca896fbb51eb6138a777ad94462147567115085e808c20a3356" gracePeriod=30
Jul 30 00:12:50.742926 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:50.742896    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"
Jul 30 00:12:50.896808 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:50.896723    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f759b397-beac-4707-ba7f-1c1bb88128c1-cert\") pod \"f759b397-beac-4707-ba7f-1c1bb88128c1\" (UID: \"f759b397-beac-4707-ba7f-1c1bb88128c1\") "
Jul 30 00:12:50.896808 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:50.896804    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sqg9\" (UniqueName: \"kubernetes.io/projected/f759b397-beac-4707-ba7f-1c1bb88128c1-kube-api-access-5sqg9\") pod \"f759b397-beac-4707-ba7f-1c1bb88128c1\" (UID: \"f759b397-beac-4707-ba7f-1c1bb88128c1\") "
Jul 30 00:12:50.898829 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:50.898803    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f759b397-beac-4707-ba7f-1c1bb88128c1-cert" (OuterVolumeSpecName: "cert") pod "f759b397-beac-4707-ba7f-1c1bb88128c1" (UID: "f759b397-beac-4707-ba7f-1c1bb88128c1"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:12:50.898943 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:50.898925    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f759b397-beac-4707-ba7f-1c1bb88128c1-kube-api-access-5sqg9" (OuterVolumeSpecName: "kube-api-access-5sqg9") pod "f759b397-beac-4707-ba7f-1c1bb88128c1" (UID: "f759b397-beac-4707-ba7f-1c1bb88128c1"). InnerVolumeSpecName "kube-api-access-5sqg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:12:50.998050 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:50.998012    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sqg9\" (UniqueName: \"kubernetes.io/projected/f759b397-beac-4707-ba7f-1c1bb88128c1-kube-api-access-5sqg9\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:12:50.998050 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:50.998042    2569 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f759b397-beac-4707-ba7f-1c1bb88128c1-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:12:51.480914 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:51.480876    2569 generic.go:356] "Generic (PLEG): container finished" podID="f759b397-beac-4707-ba7f-1c1bb88128c1" containerID="0f46d27ecbfd6ca896fbb51eb6138a777ad94462147567115085e808c20a3356" exitCode=0
Jul 30 00:12:51.481092 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:51.480918    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"
Jul 30 00:12:51.481092 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:51.480952    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l" event={"ID":"f759b397-beac-4707-ba7f-1c1bb88128c1","Type":"ContainerDied","Data":"0f46d27ecbfd6ca896fbb51eb6138a777ad94462147567115085e808c20a3356"}
Jul 30 00:12:51.481092 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:51.481021    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l" event={"ID":"f759b397-beac-4707-ba7f-1c1bb88128c1","Type":"ContainerDied","Data":"80385aed1935413d7012f0421e6efa4ed94f2c0e97da3ce5cc1135dabd393f54"}
Jul 30 00:12:51.481092 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:51.481049    2569 scope.go:117] "RemoveContainer" containerID="0f46d27ecbfd6ca896fbb51eb6138a777ad94462147567115085e808c20a3356"
Jul 30 00:12:51.493178 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:51.493159    2569 scope.go:117] "RemoveContainer" containerID="0f46d27ecbfd6ca896fbb51eb6138a777ad94462147567115085e808c20a3356"
Jul 30 00:12:51.493578 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:12:51.493546    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f46d27ecbfd6ca896fbb51eb6138a777ad94462147567115085e808c20a3356\": container with ID starting with 0f46d27ecbfd6ca896fbb51eb6138a777ad94462147567115085e808c20a3356 not found: ID does not exist" containerID="0f46d27ecbfd6ca896fbb51eb6138a777ad94462147567115085e808c20a3356"
Jul 30 00:12:51.493684 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:51.493585    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f46d27ecbfd6ca896fbb51eb6138a777ad94462147567115085e808c20a3356"} err="failed to get container status \"0f46d27ecbfd6ca896fbb51eb6138a777ad94462147567115085e808c20a3356\": rpc error: code = NotFound desc = could not find container \"0f46d27ecbfd6ca896fbb51eb6138a777ad94462147567115085e808c20a3356\": container with ID starting with 0f46d27ecbfd6ca896fbb51eb6138a777ad94462147567115085e808c20a3356 not found: ID does not exist"
Jul 30 00:12:51.495384 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:51.495363    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"]
Jul 30 00:12:51.498784 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:51.498764    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-68fcfb49c7-g5w7l"]
Jul 30 00:12:53.474004 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:12:53.473971    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f759b397-beac-4707-ba7f-1c1bb88128c1" path="/var/lib/kubelet/pods/f759b397-beac-4707-ba7f-1c1bb88128c1/volumes"
Jul 30 00:13:33.818260 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:33.818168    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bf526/must-gather-877bs"]
Jul 30 00:13:33.818738 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:33.818508    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f759b397-beac-4707-ba7f-1c1bb88128c1" containerName="manager"
Jul 30 00:13:33.818738 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:33.818519    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f759b397-beac-4707-ba7f-1c1bb88128c1" containerName="manager"
Jul 30 00:13:33.818738 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:33.818569    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f759b397-beac-4707-ba7f-1c1bb88128c1" containerName="manager"
Jul 30 00:13:33.821552 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:33.821527    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bf526/must-gather-877bs"
Jul 30 00:13:33.823808 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:33.823784    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bf526\"/\"kube-root-ca.crt\""
Jul 30 00:13:33.824341 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:33.824322    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bf526\"/\"default-dockercfg-cqmck\""
Jul 30 00:13:33.824427 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:33.824322    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bf526\"/\"openshift-service-ca.crt\""
Jul 30 00:13:33.830337 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:33.830305    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bf526/must-gather-877bs"]
Jul 30 00:13:33.855682 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:33.855644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2thm\" (UniqueName: \"kubernetes.io/projected/412cac4f-77ae-4a72-8c29-10cc3202ed8d-kube-api-access-h2thm\") pod \"must-gather-877bs\" (UID: \"412cac4f-77ae-4a72-8c29-10cc3202ed8d\") " pod="openshift-must-gather-bf526/must-gather-877bs"
Jul 30 00:13:33.855858 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:33.855710    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/412cac4f-77ae-4a72-8c29-10cc3202ed8d-must-gather-output\") pod \"must-gather-877bs\" (UID: \"412cac4f-77ae-4a72-8c29-10cc3202ed8d\") " pod="openshift-must-gather-bf526/must-gather-877bs"
Jul 30 00:13:33.956764 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:33.956725    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/412cac4f-77ae-4a72-8c29-10cc3202ed8d-must-gather-output\") pod \"must-gather-877bs\" (UID: \"412cac4f-77ae-4a72-8c29-10cc3202ed8d\") " pod="openshift-must-gather-bf526/must-gather-877bs"
Jul 30 00:13:33.956952 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:33.956822    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2thm\" (UniqueName: \"kubernetes.io/projected/412cac4f-77ae-4a72-8c29-10cc3202ed8d-kube-api-access-h2thm\") pod \"must-gather-877bs\" (UID: \"412cac4f-77ae-4a72-8c29-10cc3202ed8d\") " pod="openshift-must-gather-bf526/must-gather-877bs"
Jul 30 00:13:33.957225 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:33.957203    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/412cac4f-77ae-4a72-8c29-10cc3202ed8d-must-gather-output\") pod \"must-gather-877bs\" (UID: \"412cac4f-77ae-4a72-8c29-10cc3202ed8d\") " pod="openshift-must-gather-bf526/must-gather-877bs"
Jul 30 00:13:33.964196 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:33.964168    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2thm\" (UniqueName: \"kubernetes.io/projected/412cac4f-77ae-4a72-8c29-10cc3202ed8d-kube-api-access-h2thm\") pod \"must-gather-877bs\" (UID: \"412cac4f-77ae-4a72-8c29-10cc3202ed8d\") " pod="openshift-must-gather-bf526/must-gather-877bs"
Jul 30 00:13:34.131989 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:34.131893    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bf526/must-gather-877bs"
Jul 30 00:13:34.263158 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:34.263124    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bf526/must-gather-877bs"]
Jul 30 00:13:34.266229 ip-10-0-135-98 kubenswrapper[2569]: W0730 00:13:34.266200    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod412cac4f_77ae_4a72_8c29_10cc3202ed8d.slice/crio-c69cab7acadc34536eadd711c4bc546a7c630ec6b25b385cf3a3da5d2b846b4f WatchSource:0}: Error finding container c69cab7acadc34536eadd711c4bc546a7c630ec6b25b385cf3a3da5d2b846b4f: Status 404 returned error can't find the container with id c69cab7acadc34536eadd711c4bc546a7c630ec6b25b385cf3a3da5d2b846b4f
Jul 30 00:13:34.656898 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:34.656856    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bf526/must-gather-877bs" event={"ID":"412cac4f-77ae-4a72-8c29-10cc3202ed8d","Type":"ContainerStarted","Data":"c69cab7acadc34536eadd711c4bc546a7c630ec6b25b385cf3a3da5d2b846b4f"}
Jul 30 00:13:41.694068 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:41.694024    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bf526/must-gather-877bs" event={"ID":"412cac4f-77ae-4a72-8c29-10cc3202ed8d","Type":"ContainerStarted","Data":"595658506c8175f32f2f7900f447c60a4bc23fedf1409fe9dfabc7ccf787f06e"}
Jul 30 00:13:41.694683 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:41.694657    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bf526/must-gather-877bs" event={"ID":"412cac4f-77ae-4a72-8c29-10cc3202ed8d","Type":"ContainerStarted","Data":"00234318db7dec00db46aefff76097411aa145fad032743d944adc9a40251e90"}
Jul 30 00:13:41.709157 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:13:41.709093    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bf526/must-gather-877bs" podStartSLOduration=2.197707215 podStartE2EDuration="8.709069339s" totalImagesPullingTime="6.511362124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:13:41.708141044 +0000 UTC m=+5228.713338524" watchObservedRunningTime="2026-07-30 00:13:41.709069339 +0000 UTC m=+5228.714266764"
Jul 30 00:14:00.936193 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:00.936159    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:02.113550 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:02.113517    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:03.743975 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:03.743946    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:04.763837 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:04.763807    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:05.792199 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:05.792167    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:06.825130 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:06.825095    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:07.820379 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:07.820349    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:08.823317 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:08.823288    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:09.840356 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:09.840320    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:10.847827 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:10.847800    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:11.885036 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:11.885009    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:12.909366 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:12.909337    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:13.957725 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:13.957689    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:14.971934 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:14.971906    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:16.011714 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:16.011686    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:17.024003 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:17.023977    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:18.043557 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:18.043525    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:19.106881 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:19.106851    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:20.116622 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:20.116593    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:21.126002 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:21.125975    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:22.172086 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:22.172056    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:23.233625 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:23.233595    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:24.259497 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:24.259455    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:25.292189 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:25.292156    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:26.309132 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:26.309105    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:27.904538 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:27.904505    2569 generic.go:356] "Generic (PLEG): container finished" podID="412cac4f-77ae-4a72-8c29-10cc3202ed8d" containerID="00234318db7dec00db46aefff76097411aa145fad032743d944adc9a40251e90" exitCode=0
Jul 30 00:14:27.905253 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:27.904554    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bf526/must-gather-877bs" event={"ID":"412cac4f-77ae-4a72-8c29-10cc3202ed8d","Type":"ContainerDied","Data":"00234318db7dec00db46aefff76097411aa145fad032743d944adc9a40251e90"}
Jul 30 00:14:27.905253 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:27.904892    2569 scope.go:117] "RemoveContainer" containerID="00234318db7dec00db46aefff76097411aa145fad032743d944adc9a40251e90"
Jul 30 00:14:27.987985 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:27.987944    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bf526_must-gather-877bs_412cac4f-77ae-4a72-8c29-10cc3202ed8d/gather/0.log"
Jul 30 00:14:31.420603 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:31.420571    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w9w6m_450449c0-a316-47ca-81cd-844420f36bea/global-pull-secret-syncer/0.log"
Jul 30 00:14:31.580314 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:31.580281    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r4lqd_f678c5dc-4755-4702-9be8-79557973646a/konnectivity-agent/0.log"
Jul 30 00:14:31.655753 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:31.655722    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-98.ec2.internal_c3fee7606f52b4c69acd2656db749d26/haproxy/0.log"
Jul 30 00:14:33.398251 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.398209    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-bf526/must-gather-877bs"]
Jul 30 00:14:33.398778 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.398439    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-bf526/must-gather-877bs" podUID="412cac4f-77ae-4a72-8c29-10cc3202ed8d" containerName="copy" containerID="cri-o://595658506c8175f32f2f7900f447c60a4bc23fedf1409fe9dfabc7ccf787f06e" gracePeriod=2
Jul 30 00:14:33.400658 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:14:33.400607    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-877bs\" is forbidden: User \"system:node:ip-10-0-135-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bf526\": no relationship found between node 'ip-10-0-135-98.ec2.internal' and this object" podUID="412cac4f-77ae-4a72-8c29-10cc3202ed8d" pod="openshift-must-gather-bf526/must-gather-877bs"
Jul 30 00:14:33.401063 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.401042    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-bf526/must-gather-877bs"]
Jul 30 00:14:33.472967 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:14:33.472905    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-877bs\" is forbidden: User \"system:node:ip-10-0-135-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bf526\": no relationship found between node 'ip-10-0-135-98.ec2.internal' and this object" podUID="412cac4f-77ae-4a72-8c29-10cc3202ed8d" pod="openshift-must-gather-bf526/must-gather-877bs"
Jul 30 00:14:33.638584 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.638560    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bf526_must-gather-877bs_412cac4f-77ae-4a72-8c29-10cc3202ed8d/copy/0.log"
Jul 30 00:14:33.641334 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.641312    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bf526/must-gather-877bs"
Jul 30 00:14:33.707564 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.707449    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/412cac4f-77ae-4a72-8c29-10cc3202ed8d-must-gather-output\") pod \"412cac4f-77ae-4a72-8c29-10cc3202ed8d\" (UID: \"412cac4f-77ae-4a72-8c29-10cc3202ed8d\") "
Jul 30 00:14:33.707564 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.707525    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2thm\" (UniqueName: \"kubernetes.io/projected/412cac4f-77ae-4a72-8c29-10cc3202ed8d-kube-api-access-h2thm\") pod \"412cac4f-77ae-4a72-8c29-10cc3202ed8d\" (UID: \"412cac4f-77ae-4a72-8c29-10cc3202ed8d\") "
Jul 30 00:14:33.709804 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.709770    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412cac4f-77ae-4a72-8c29-10cc3202ed8d-kube-api-access-h2thm" (OuterVolumeSpecName: "kube-api-access-h2thm") pod "412cac4f-77ae-4a72-8c29-10cc3202ed8d" (UID: "412cac4f-77ae-4a72-8c29-10cc3202ed8d"). InnerVolumeSpecName "kube-api-access-h2thm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:14:33.712498 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.712460    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412cac4f-77ae-4a72-8c29-10cc3202ed8d-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "412cac4f-77ae-4a72-8c29-10cc3202ed8d" (UID: "412cac4f-77ae-4a72-8c29-10cc3202ed8d"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:14:33.808207 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.808174    2569 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/412cac4f-77ae-4a72-8c29-10cc3202ed8d-must-gather-output\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:14:33.808207 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.808202    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2thm\" (UniqueName: \"kubernetes.io/projected/412cac4f-77ae-4a72-8c29-10cc3202ed8d-kube-api-access-h2thm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Jul 30 00:14:33.928943 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.928913    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bf526_must-gather-877bs_412cac4f-77ae-4a72-8c29-10cc3202ed8d/copy/0.log"
Jul 30 00:14:33.929310 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.929289    2569 generic.go:356] "Generic (PLEG): container finished" podID="412cac4f-77ae-4a72-8c29-10cc3202ed8d" containerID="595658506c8175f32f2f7900f447c60a4bc23fedf1409fe9dfabc7ccf787f06e" exitCode=143
Jul 30 00:14:33.929402 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.929352    2569 scope.go:117] "RemoveContainer" containerID="595658506c8175f32f2f7900f447c60a4bc23fedf1409fe9dfabc7ccf787f06e"
Jul 30 00:14:33.929457 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.929346    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bf526/must-gather-877bs"
Jul 30 00:14:33.938581 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.938366    2569 scope.go:117] "RemoveContainer" containerID="00234318db7dec00db46aefff76097411aa145fad032743d944adc9a40251e90"
Jul 30 00:14:33.952208 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.952186    2569 scope.go:117] "RemoveContainer" containerID="595658506c8175f32f2f7900f447c60a4bc23fedf1409fe9dfabc7ccf787f06e"
Jul 30 00:14:33.952577 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:14:33.952475    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"595658506c8175f32f2f7900f447c60a4bc23fedf1409fe9dfabc7ccf787f06e\": container with ID starting with 595658506c8175f32f2f7900f447c60a4bc23fedf1409fe9dfabc7ccf787f06e not found: ID does not exist" containerID="595658506c8175f32f2f7900f447c60a4bc23fedf1409fe9dfabc7ccf787f06e"
Jul 30 00:14:33.952577 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.952534    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"595658506c8175f32f2f7900f447c60a4bc23fedf1409fe9dfabc7ccf787f06e"} err="failed to get container status \"595658506c8175f32f2f7900f447c60a4bc23fedf1409fe9dfabc7ccf787f06e\": rpc error: code = NotFound desc = could not find container \"595658506c8175f32f2f7900f447c60a4bc23fedf1409fe9dfabc7ccf787f06e\": container with ID starting with 595658506c8175f32f2f7900f447c60a4bc23fedf1409fe9dfabc7ccf787f06e not found: ID does not exist"
Jul 30 00:14:33.952577 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.952555    2569 scope.go:117] "RemoveContainer" containerID="00234318db7dec00db46aefff76097411aa145fad032743d944adc9a40251e90"
Jul 30 00:14:33.952852 ip-10-0-135-98 kubenswrapper[2569]: E0730 00:14:33.952833    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00234318db7dec00db46aefff76097411aa145fad032743d944adc9a40251e90\": container with ID starting with 00234318db7dec00db46aefff76097411aa145fad032743d944adc9a40251e90 not found: ID does not exist" containerID="00234318db7dec00db46aefff76097411aa145fad032743d944adc9a40251e90"
Jul 30 00:14:33.952889 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:33.952860    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00234318db7dec00db46aefff76097411aa145fad032743d944adc9a40251e90"} err="failed to get container status \"00234318db7dec00db46aefff76097411aa145fad032743d944adc9a40251e90\": rpc error: code = NotFound desc = could not find container \"00234318db7dec00db46aefff76097411aa145fad032743d944adc9a40251e90\": container with ID starting with 00234318db7dec00db46aefff76097411aa145fad032743d944adc9a40251e90 not found: ID does not exist"
Jul 30 00:14:35.474400 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:35.474357    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="412cac4f-77ae-4a72-8c29-10cc3202ed8d" path="/var/lib/kubelet/pods/412cac4f-77ae-4a72-8c29-10cc3202ed8d/volumes"
Jul 30 00:14:37.365644 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:37.365613    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vrnns_7a212016-b3bc-4ad1-8b8d-ce9134050cfe/kuadrant-console-plugin/0.log"
Jul 30 00:14:39.115264 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:39.115231    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-lzctb_8088e072-67e5-4ff4-a931-e881cb0e69fa/kube-state-metrics/0.log"
Jul 30 00:14:39.147200 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:39.147167    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-lzctb_8088e072-67e5-4ff4-a931-e881cb0e69fa/kube-rbac-proxy-main/0.log"
Jul 30 00:14:39.193207 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:39.193172    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-lzctb_8088e072-67e5-4ff4-a931-e881cb0e69fa/kube-rbac-proxy-self/0.log"
Jul 30 00:14:39.227128 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:39.227087    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bb87447bb-z4kjc_275a9a4a-e06e-40a9-9c9e-45562d22fcbd/metrics-server/0.log"
Jul 30 00:14:39.467577 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:39.467533    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kn7nm_704b9290-51eb-4000-9ab2-7ae30d6f8764/node-exporter/0.log"
Jul 30 00:14:39.492574 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:39.492540    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kn7nm_704b9290-51eb-4000-9ab2-7ae30d6f8764/kube-rbac-proxy/0.log"
Jul 30 00:14:39.522293 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:39.522268    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kn7nm_704b9290-51eb-4000-9ab2-7ae30d6f8764/init-textfile/0.log"
Jul 30 00:14:39.921170 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:39.921089    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5df795b4f-bxd4s_35208e50-6550-46e0-8c9d-6e25ddf655ff/telemeter-client/0.log"
Jul 30 00:14:39.946794 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:39.946766    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5df795b4f-bxd4s_35208e50-6550-46e0-8c9d-6e25ddf655ff/reload/0.log"
Jul 30 00:14:39.971200 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:39.971174    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5df795b4f-bxd4s_35208e50-6550-46e0-8c9d-6e25ddf655ff/kube-rbac-proxy/0.log"
Jul 30 00:14:41.575652 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.575616    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/2.log"
Jul 30 00:14:41.580188 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.580154    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-27xfm_c0f949f6-90cf-4e28-a713-3587649d077c/console-operator/3.log"
Jul 30 00:14:41.847768 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.847686    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"]
Jul 30 00:14:41.848006 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.847993    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412cac4f-77ae-4a72-8c29-10cc3202ed8d" containerName="copy"
Jul 30 00:14:41.848049 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.848007    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="412cac4f-77ae-4a72-8c29-10cc3202ed8d" containerName="copy"
Jul 30 00:14:41.848049 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.848017    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412cac4f-77ae-4a72-8c29-10cc3202ed8d" containerName="gather"
Jul 30 00:14:41.848049 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.848023    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="412cac4f-77ae-4a72-8c29-10cc3202ed8d" containerName="gather"
Jul 30 00:14:41.848149 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.848073    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="412cac4f-77ae-4a72-8c29-10cc3202ed8d" containerName="gather"
Jul 30 00:14:41.848149 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.848082    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="412cac4f-77ae-4a72-8c29-10cc3202ed8d" containerName="copy"
Jul 30 00:14:41.853649 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.853623    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:41.856040 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.856007    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6zt68\"/\"kube-root-ca.crt\""
Jul 30 00:14:41.856040 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.856024    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6zt68\"/\"default-dockercfg-k9xqk\""
Jul 30 00:14:41.856255 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.856154    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6zt68\"/\"openshift-service-ca.crt\""
Jul 30 00:14:41.857519 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.857460    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"]
Jul 30 00:14:41.977539 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.977472    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ncxt\" (UniqueName: \"kubernetes.io/projected/4a69679e-8708-4fee-adf8-a11c5064a28f-kube-api-access-9ncxt\") pod \"perf-node-gather-daemonset-9r42h\" (UID: \"4a69679e-8708-4fee-adf8-a11c5064a28f\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:41.977741 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.977574    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a69679e-8708-4fee-adf8-a11c5064a28f-sys\") pod \"perf-node-gather-daemonset-9r42h\" (UID: \"4a69679e-8708-4fee-adf8-a11c5064a28f\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:41.977741 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.977600    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a69679e-8708-4fee-adf8-a11c5064a28f-proc\") pod \"perf-node-gather-daemonset-9r42h\" (UID: \"4a69679e-8708-4fee-adf8-a11c5064a28f\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:41.977741 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.977666    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a69679e-8708-4fee-adf8-a11c5064a28f-lib-modules\") pod \"perf-node-gather-daemonset-9r42h\" (UID: \"4a69679e-8708-4fee-adf8-a11c5064a28f\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:41.977741 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:41.977703    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a69679e-8708-4fee-adf8-a11c5064a28f-podres\") pod \"perf-node-gather-daemonset-9r42h\" (UID: \"4a69679e-8708-4fee-adf8-a11c5064a28f\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:42.079129 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.079092    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a69679e-8708-4fee-adf8-a11c5064a28f-sys\") pod \"perf-node-gather-daemonset-9r42h\" (UID: \"4a69679e-8708-4fee-adf8-a11c5064a28f\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:42.079129 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.079137    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a69679e-8708-4fee-adf8-a11c5064a28f-proc\") pod \"perf-node-gather-daemonset-9r42h\" (UID: \"4a69679e-8708-4fee-adf8-a11c5064a28f\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:42.079353 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.079188    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a69679e-8708-4fee-adf8-a11c5064a28f-lib-modules\") pod \"perf-node-gather-daemonset-9r42h\" (UID: \"4a69679e-8708-4fee-adf8-a11c5064a28f\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:42.079353 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.079214    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a69679e-8708-4fee-adf8-a11c5064a28f-podres\") pod \"perf-node-gather-daemonset-9r42h\" (UID: \"4a69679e-8708-4fee-adf8-a11c5064a28f\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:42.079353 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.079235    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a69679e-8708-4fee-adf8-a11c5064a28f-sys\") pod \"perf-node-gather-daemonset-9r42h\" (UID: \"4a69679e-8708-4fee-adf8-a11c5064a28f\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:42.079353 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.079244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ncxt\" (UniqueName: \"kubernetes.io/projected/4a69679e-8708-4fee-adf8-a11c5064a28f-kube-api-access-9ncxt\") pod \"perf-node-gather-daemonset-9r42h\" (UID: \"4a69679e-8708-4fee-adf8-a11c5064a28f\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:42.079353 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.079301    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a69679e-8708-4fee-adf8-a11c5064a28f-proc\") pod \"perf-node-gather-daemonset-9r42h\" (UID: \"4a69679e-8708-4fee-adf8-a11c5064a28f\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:42.079353 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.079342    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a69679e-8708-4fee-adf8-a11c5064a28f-lib-modules\") pod \"perf-node-gather-daemonset-9r42h\" (UID: \"4a69679e-8708-4fee-adf8-a11c5064a28f\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:42.079353 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.079345    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a69679e-8708-4fee-adf8-a11c5064a28f-podres\") pod \"perf-node-gather-daemonset-9r42h\" (UID: \"4a69679e-8708-4fee-adf8-a11c5064a28f\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:42.086396 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.086360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ncxt\" (UniqueName: \"kubernetes.io/projected/4a69679e-8708-4fee-adf8-a11c5064a28f-kube-api-access-9ncxt\") pod \"perf-node-gather-daemonset-9r42h\" (UID: \"4a69679e-8708-4fee-adf8-a11c5064a28f\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:42.117407 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.117331    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-4qmvc_5e2c4d0c-99fa-45cd-b046-82a40d8a97a0/download-server/0.log"
Jul 30 00:14:42.171517 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.171463    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:42.300545 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.300397    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"]
Jul 30 00:14:42.303258 ip-10-0-135-98 kubenswrapper[2569]: W0730 00:14:42.303227    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4a69679e_8708_4fee_adf8_a11c5064a28f.slice/crio-91aecb5d31dae6849e4c475919d2baffd03c4564859535e8613df5c258b3da6c WatchSource:0}: Error finding container 91aecb5d31dae6849e4c475919d2baffd03c4564859535e8613df5c258b3da6c: Status 404 returned error can't find the container with id 91aecb5d31dae6849e4c475919d2baffd03c4564859535e8613df5c258b3da6c
Jul 30 00:14:42.967218 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.967174    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h" event={"ID":"4a69679e-8708-4fee-adf8-a11c5064a28f","Type":"ContainerStarted","Data":"86c147336314c207574761976970f545ca441d5edffb4a0475581d712da31140"}
Jul 30 00:14:42.967666 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.967241    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h" event={"ID":"4a69679e-8708-4fee-adf8-a11c5064a28f","Type":"ContainerStarted","Data":"91aecb5d31dae6849e4c475919d2baffd03c4564859535e8613df5c258b3da6c"}
Jul 30 00:14:42.967666 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.967280    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:42.981716 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:42.981666    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h" podStartSLOduration=1.981650597 podStartE2EDuration="1.981650597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:14:42.980760167 +0000 UTC m=+5289.985957597" watchObservedRunningTime="2026-07-30 00:14:42.981650597 +0000 UTC m=+5289.986848024"
Jul 30 00:14:43.767236 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:43.767204    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xxsch_49ebc216-8a22-460c-b045-06da7890eba1/dns/0.log"
Jul 30 00:14:43.790461 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:43.790429    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xxsch_49ebc216-8a22-460c-b045-06da7890eba1/kube-rbac-proxy/0.log"
Jul 30 00:14:43.815320 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:43.815295    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mbdtd_06f404ba-6594-4814-b5c5-9904a205207c/dns-node-resolver/0.log"
Jul 30 00:14:44.431547 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:44.431509    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7778c94ff4-8ns4j_bbd2381d-ee5b-4a80-9d71-a98c3866ed4f/registry/0.log"
Jul 30 00:14:44.506643 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:44.506612    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r8cns_8402f600-296c-4fa7-a6a2-1d34bb23781c/node-ca/0.log"
Jul 30 00:14:46.314208 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:46.314181    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wf4k2_bc372877-f198-4771-b46c-8ebd74cca4bc/serve-healthcheck-canary/0.log"
Jul 30 00:14:46.804877 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:46.804847    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5d2kg_1bf23056-3f7e-414c-917a-1261f7310bdb/kube-rbac-proxy/0.log"
Jul 30 00:14:46.827403 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:46.827380    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5d2kg_1bf23056-3f7e-414c-917a-1261f7310bdb/exporter/0.log"
Jul 30 00:14:46.851464 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:46.851437    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5d2kg_1bf23056-3f7e-414c-917a-1261f7310bdb/extractor/0.log"
Jul 30 00:14:48.980779 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:48.980732    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-9r42h"
Jul 30 00:14:49.900499 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:49.900463    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-78dcd69c74-5zff2_4fc1b98b-b6e0-482b-afad-dd4c0db5d603/manager/0.log"
Jul 30 00:14:50.649975 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:50.649939    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-7586c4769-btsn4_f28c7d91-fe2f-4c45-a927-22fb7b3aac6b/manager/0.log"
Jul 30 00:14:51.217241 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:51.217205    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-mhzx7_1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1/s3-init/0.log"
Jul 30 00:14:51.238657 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:51.238632    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-mhzx7_1145b1ed-c80c-4b4a-b3a8-bf3f98ff8ba1/download-hf-model/0.log"
Jul 30 00:14:57.925650 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:57.925563    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-shgn2_5ab88d84-44ee-463c-986e-43d1da51284b/kube-multus-additional-cni-plugins/0.log"
Jul 30 00:14:57.949966 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:57.949939    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-shgn2_5ab88d84-44ee-463c-986e-43d1da51284b/egress-router-binary-copy/0.log"
Jul 30 00:14:57.973966 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:57.973937    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-shgn2_5ab88d84-44ee-463c-986e-43d1da51284b/cni-plugins/0.log"
Jul 30 00:14:57.997831 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:57.997803    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-shgn2_5ab88d84-44ee-463c-986e-43d1da51284b/bond-cni-plugin/0.log"
Jul 30 00:14:58.023685 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:58.023654    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-shgn2_5ab88d84-44ee-463c-986e-43d1da51284b/routeoverride-cni/0.log"
Jul 30 00:14:58.050979 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:58.050946    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-shgn2_5ab88d84-44ee-463c-986e-43d1da51284b/whereabouts-cni-bincopy/0.log"
Jul 30 00:14:58.074954 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:58.074923    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-shgn2_5ab88d84-44ee-463c-986e-43d1da51284b/whereabouts-cni/0.log"
Jul 30 00:14:58.279905 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:58.279858    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fxc56_6289a7d5-1978-462d-9ffc-a4135ff29c6d/kube-multus/0.log"
Jul 30 00:14:58.386938 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:58.386914    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t5x5r_0d19dc00-e3da-4306-a5ff-8cca92e02a1c/network-metrics-daemon/0.log"
Jul 30 00:14:58.433392 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:58.433356    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t5x5r_0d19dc00-e3da-4306-a5ff-8cca92e02a1c/kube-rbac-proxy/0.log"
Jul 30 00:14:59.754861 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:59.754830    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvw95_9e4e9b57-17f8-495a-b460-319701b58ba3/ovn-controller/0.log"
Jul 30 00:14:59.773863 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:59.773832    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvw95_9e4e9b57-17f8-495a-b460-319701b58ba3/ovn-acl-logging/0.log"
Jul 30 00:14:59.802735 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:59.802704    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvw95_9e4e9b57-17f8-495a-b460-319701b58ba3/kube-rbac-proxy-node/0.log"
Jul 30 00:14:59.825307 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:59.825278    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvw95_9e4e9b57-17f8-495a-b460-319701b58ba3/kube-rbac-proxy-ovn-metrics/0.log"
Jul 30 00:14:59.845504 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:59.845466    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvw95_9e4e9b57-17f8-495a-b460-319701b58ba3/northd/0.log"
Jul 30 00:14:59.868128 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:59.868105    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvw95_9e4e9b57-17f8-495a-b460-319701b58ba3/nbdb/0.log"
Jul 30 00:14:59.890827 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:59.890801    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvw95_9e4e9b57-17f8-495a-b460-319701b58ba3/sbdb/0.log"
Jul 30 00:14:59.954174 ip-10-0-135-98 kubenswrapper[2569]: I0730 00:14:59.954139    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvw95_9e4e9b57-17f8-495a-b460-319701b58ba3/ovnkube-controller/0.log"