Jul 29 10:28:51.698886 ip-10-0-140-240 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 10:28:52.065168 ip-10-0-140-240 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:28:52.065168 ip-10-0-140-240 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 10:28:52.065168 ip-10-0-140-240 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:28:52.065168 ip-10-0-140-240 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 10:28:52.065168 ip-10-0-140-240 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:28:52.066106 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.066017    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 10:28:52.071456 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071439    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:52.071456 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071455    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071459    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071463    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071467    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071469    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071473    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071476    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071478    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071481    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071484    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071487    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071489    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071492    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071495    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071497    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071500    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071503    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071505    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071508    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071517    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:52.071530 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071520    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071522    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071525    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071529    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071533    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071538    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071542    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071546    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071548    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071551    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071553    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071556    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071559    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071561    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071564    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071567    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071569    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071572    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071575    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:52.072027 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071578    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071581    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071583    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071586    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071588    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071592    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071595    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071598    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071600    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071603    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071607    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071609    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071612    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071614    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071617    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071620    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071623    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071626    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071635    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071638    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:52.072486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071641    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071643    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071646    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071649    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071651    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071654    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071657    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071659    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071662    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071664    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071667    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071670    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071685    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071687    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071691    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071693    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071697    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071700    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071703    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:52.073007 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071706    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071708    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071711    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071713    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071717    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071720    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071722    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071725    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071728    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071731    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071734    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071737    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071739    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071742    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071745    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071747    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071750    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071754    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071756    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.071759    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:52.073458 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072492    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072499    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072502    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072505    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072508    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072524    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072526    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072529    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072532    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072534    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072537    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072540    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072542    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072545    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072548    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072550    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072553    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072556    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072559    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:52.073980 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072561    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072564    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072567    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072569    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072572    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072575    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072577    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072580    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072582    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072586    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072590    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072593    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072596    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072599    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072601    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072604    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072607    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072610    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072613    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:52.074454 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072624    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072627    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072629    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072632    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072634    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072637    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072639    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072642    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072645    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072647    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072653    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072656    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072659    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072661    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072665    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072667    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072672    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072691    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072695    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072697    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:52.075044 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072700    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072703    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072705    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072708    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072710    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072713    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072716    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072720    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072723    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072726    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072729    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072732    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072734    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072745    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072748    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072751    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072753    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072756    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072759    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:52.075526 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072762    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072764    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072767    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072771    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072773    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072776    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072778    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072781    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072783    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072786    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072788    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072791    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072794    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072796    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072798    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072801    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072804    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072807    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072810    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072812    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:52.076009 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072814    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.072817    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073205    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073210    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073214    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073217    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073222    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073224    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073227    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073230    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073233    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073235    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073238    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073241    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073244    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073247    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073250    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073253    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073255    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073258    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:52.076495 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073260    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073265    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073268    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073271    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073274    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073277    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073280    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073283    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073286    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073288    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073291    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073294    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073297    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073300    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073302    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073305    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073308    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073310    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073313    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:52.077002 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073316    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073319    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073321    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073324    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073327    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073330    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073332    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073335    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073339    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073341    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073344    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073346    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073349    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073351    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073354    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073356    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073359    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073361    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073364    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073366    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:52.077497 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073369    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073371    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073374    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073376    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073379    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073381    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073384    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073387    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073389    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073392    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073394    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073397    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073400    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073402    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073405    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073407    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073410    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073412    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073415    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:52.078054 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073419    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073424    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073427    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073430    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073433    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073436    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073439    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073442    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073444    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073447    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073450    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073452    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073455    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073457    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073460    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073463    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073465    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073468    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073471    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073474    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:52.078514 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073476    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073479    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.073482    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073557    2584 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073565    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073571    2584 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073576    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073581    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073584    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073589    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073594    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073597    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073600    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073605    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073609    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073612    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073615    2584 flags.go:64] FLAG: --cgroup-root=""
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073618    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073621    2584 flags.go:64] FLAG: --client-ca-file=""
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073624    2584 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073627    2584 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073630    2584 flags.go:64] FLAG: --cluster-domain=""
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073633    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 10:28:52.079010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073636    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073640    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073643    2584 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073647    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073650    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073653    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073657    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073660    2584 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073663    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073666    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073669    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073672    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073693    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073698    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073702    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073706    2584 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073709    2584 flags.go:64] FLAG: --enable-server="true"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073713    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073718    2584 flags.go:64] FLAG: --event-burst="100"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073721    2584 flags.go:64] FLAG: --event-qps="50"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073724    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073727    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073732    2584 flags.go:64] FLAG: --eviction-hard=""
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073736    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 10:28:52.079556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073739    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073742    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073745    2584 flags.go:64] FLAG: --eviction-soft=""
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073748    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073751    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073754    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073757    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073760    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073763    2584 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073766    2584 flags.go:64] FLAG: --feature-gates=""
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073770    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073773    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073776    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073779    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073782    2584 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073785    2584 flags.go:64] FLAG: --help="false"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073788    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073791    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073794    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073799    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073803    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073806    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073809    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 10:28:52.080153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073812    2584 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073815    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073818    2584 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073821    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073825    2584 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073827    2584 flags.go:64] FLAG: --kube-reserved=""
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073830    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073835    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073839    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073841    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073844    2584 flags.go:64] FLAG: --lock-file=""
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073847    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073850    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073854    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073860    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073863    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073866    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073869    2584 flags.go:64] FLAG: --logging-format="text"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073872    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073875    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073878    2584 flags.go:64] FLAG: --manifest-url=""
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073881    2584 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073886    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073889    2584 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073894    2584 flags.go:64] FLAG: --max-pods="110"
Jul 29 10:28:52.080780 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073897    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073900    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073903    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073907    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073910    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073913    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073916    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073924    2584 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073927    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073930    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073933    2584 flags.go:64] FLAG: --pod-cidr=""
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073940    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073946    2584 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073949    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073953    2584 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073957    2584 flags.go:64] FLAG: --port="10250"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073960    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073963    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0185ef89317c9a40b"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073966    2584 flags.go:64] FLAG: --qos-reserved=""
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073969    2584 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073972    2584 flags.go:64] FLAG: --register-node="true"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073975    2584 flags.go:64] FLAG: --register-with-taints=""
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073978    2584 flags.go:64] FLAG: --registry-burst="10"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073981    2584 flags.go:64] FLAG: --registry-qps="5"
Jul 29 10:28:52.081435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073985    2584 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073988    2584 flags.go:64] FLAG: --reserved-memory=""
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073992    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073995    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.073998    2584 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074001    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074004    2584 flags.go:64] FLAG: --runonce="false"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074007    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074010    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074013    2584 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074015    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074019    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074022    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074026    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074029    2584 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074032    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074035    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074038    2584 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074040    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074043    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074047    2584 flags.go:64] FLAG: --system-cgroups=""
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074050    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074055    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074058    2584 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074061    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 10:28:52.082047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074065    2584 flags.go:64] FLAG: --tls-min-version=""
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074067    2584 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074071    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074073    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074076    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074079    2584 flags.go:64] FLAG: --v="2"
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074083    2584 flags.go:64] FLAG: --version="false"
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074087    2584 flags.go:64] FLAG: --vmodule=""
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074091    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.074094    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074218    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074221    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074225    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074227    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074230    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074233    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074236    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074238    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074243    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074246    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074248    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074251    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:52.082650 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074253    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074256    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074258    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074261    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074263    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074266    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074269    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074271    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074274    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074276    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074279    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074283    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074287    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074290    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074293    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074295    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074298    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074301    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074304    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074306    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:52.083218 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074309    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074312    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074314    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074317    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074320    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074322    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074325    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074327    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074331    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074334    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074336    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074339    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074341    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074344    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074346    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074349    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074352    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074354    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074357    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074359    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:52.083908 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074362    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074365    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074367    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074370    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074372    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074375    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074377    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074380    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074382    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074385    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074388    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074392    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074398    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074401    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074403    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074406    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074409    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074411    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074414    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074416    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:52.085337 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074420    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074422    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074425    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074427    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074430    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074432    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074435    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074439    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074442    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074445    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074448    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074451    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074454    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074457    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074460    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074462    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074465    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074468    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:52.085948 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074471    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:52.086422 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074473    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:52.086422 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074475    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:52.086422 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074478    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:52.086422 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074480    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:52.086422 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074483    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:52.086422 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074485    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:52.086422 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074487    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:52.086422 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.074490    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:52.086422 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.075037    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:28:52.086422 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.081992    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 10:28:52.086422 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.082012    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 10:28:52.086422 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.082032    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 10:28:52.086422 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.082042    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 10:28:52.086422 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082093    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:52.086422 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082098    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082101    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082105    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082108    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082111    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082113    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082116    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082118    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082121    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082124    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082127    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082129    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082132    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082134    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082137    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082139    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082141    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082144    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082146    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082149    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:52.086900 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082151    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082153    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082158    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082161    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082164    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082167    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082169    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082172    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082177    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082180    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082183    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082186    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082188    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082191    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082194    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082197    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082199    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082202    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082204    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:52.087448 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082207    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082209    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082212    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082214    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082217    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082219    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082222    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082224    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082227    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082231    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082235    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082237    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082240    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082243    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082245    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082248    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082251    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082253    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082256    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:52.088048 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082258    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082261    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082264    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082267    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082269    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082272    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082274    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082277    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082281    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082284    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082287    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082290    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082293    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082296    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082298    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082301    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082303    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082306    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082308    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082311    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:52.088557 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082314    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082316    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082318    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082321    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082323    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082326    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082329    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082331    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082334    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082337    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082339    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082342    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082344    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082347    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082350    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082352    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082356    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082358    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082361    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:52.089360 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082363    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:52.090220 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.082368    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:28:52.090220 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082475    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:52.090220 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082480    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:52.090220 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082483    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:52.090220 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082485    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:52.090220 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082488    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:52.090220 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082491    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:52.090220 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082494    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:52.090220 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082497    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:52.090220 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082500    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:52.090220 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082502    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:52.090220 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082505    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:52.090220 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082507    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:52.090220 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082510    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:52.090220 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082513    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082516    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082518    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082521    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082523    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082526    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082529    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082531    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082534    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082537    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082539    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082542    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082544    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082547    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082549    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082552    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082555    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082557    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082560    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082562    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:52.090706 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082565    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082568    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082570    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082573    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082575    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082578    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082580    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082583    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082585    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082588    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082590    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082593    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082595    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082599    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082602    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082605    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082607    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082610    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082613    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082615    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:52.091388 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082618    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082621    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082623    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082625    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082628    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082630    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082633    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082635    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082638    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082641    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082643    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082645    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082648    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082650    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082653    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082656    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082658    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082662    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082666    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:52.092113 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082669    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082672    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082687    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082689    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082692    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082694    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082697    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082700    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082704    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082708    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082711    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082714    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082717    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082720    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082724    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082726    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082729    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082732    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082734    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:52.092633 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082737    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:52.093185 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082740    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:52.093185 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082742    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:52.093185 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082745    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:52.093185 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082748    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:52.093185 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082750    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:52.093185 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082753    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:52.093185 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:52.082755    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:52.093185 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.082760    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:28:52.093185 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.082877    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 10:28:52.093185 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.086406    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 10:28:52.093185 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.087671    2584 server.go:1019] "Starting client certificate rotation"
Jul 29 10:28:52.093185 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.087810    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 10:28:52.093185 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.088769    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 10:28:52.093185 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.090078    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 10:28:52.093571 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.091355    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 10:28:52.102689 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.102659    2584 log.go:25] "Validated CRI v1 runtime API"
Jul 29 10:28:52.104892 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.104875    2584 log.go:25] "Validated CRI v1 image API"
Jul 29 10:28:52.106429 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.106409    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 10:28:52.108921 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.108899    2584 fs.go:135] Filesystem UUIDs: map[772c92b2-920e-4cd9-b0fb-20da229aab0b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 dc785b6e-0d8d-4f1d-92fb-5a36af603405:/dev/nvme0n1p4]
Jul 29 10:28:52.109001 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.108919    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 10:28:52.114496 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.114356    2584 manager.go:216] Machine: {Timestamp:2026-07-29 10:28:52.112963903 +0000 UTC m=+0.305991244 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103602 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2df0114bf3edf63097b399e58d43b1 SystemUUID:ec2df011-4bf3-edf6-3097-b399e58d43b1 BootID:a6f3aa1e-a3ef-425e-b3c5-162f62e7aa5c Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ce:ad:72:3e:4f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ce:ad:72:3e:4f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:08:9f:bd:f9:7b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 10:28:52.114496 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.114491    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 10:28:52.114637 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.114624    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 10:28:52.117300 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.117268    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 10:28:52.117454 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.117303    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-240.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 10:28:52.118009 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.117996    2584 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 10:28:52.118044 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.118012    2584 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 10:28:52.118044 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.118020    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 10:28:52.118679 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.118666    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 10:28:52.118720 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.118672    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 10:28:52.119832 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.119819    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 10:28:52.122650 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.122638    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 10:28:52.122811 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.122800    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 10:28:52.122862 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.122833    2584 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 10:28:52.122862 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.122851    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 10:28:52.122963 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.122871    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 10:28:52.122963 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.122881    2584 kubelet.go:398] "Adding apiserver pod source"
Jul 29 10:28:52.122963 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.122891    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 10:28:52.123277 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.123264    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 10:28:52.123311 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.123284    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 10:28:52.126983 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.126957    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 10:28:52.128334 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.128317    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 10:28:52.128415 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.128349    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 10:28:52.128908 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.128886    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 10:28:52.128965 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.128912    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 10:28:52.128965 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.128919    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 10:28:52.128965 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.128925    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 10:28:52.128965 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.128933    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 10:28:52.128965 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.128939    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 10:28:52.129240 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.129044    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 10:28:52.129240 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.129056    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 10:28:52.129240 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.129063    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 10:28:52.129240 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.129069    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 10:28:52.129240 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.129078    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 10:28:52.129240 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.129086    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 10:28:52.129240 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.129110    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 10:28:52.129240 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.129115    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 10:28:52.131951 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.131934    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-240.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 10:28:52.131951 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.131938    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 10:28:52.132234 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.132213    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-240.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 10:28:52.132784 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.132768    2584 server.go:1268] "Started kubelet"
Jul 29 10:28:52.132914 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.132883    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 10:28:52.133392 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.133348    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 10:28:52.133491 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.133437    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 10:28:52.134199 ip-10-0-140-240 systemd[1]: Started Kubernetes Kubelet.
Jul 29 10:28:52.135238 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.135151    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 10:28:52.135394 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.135381    2584 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 10:28:52.137495 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.137476    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 10:28:52.137646 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.136737    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-240.ec2.internal.18c6bbedc1d70ace  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-240.ec2.internal,UID:ip-10-0-140-240.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-240.ec2.internal,},FirstTimestamp:2026-07-29 10:28:52.13271931 +0000 UTC m=+0.325746652,LastTimestamp:2026-07-29 10:28:52.13271931 +0000 UTC m=+0.325746652,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-240.ec2.internal,}"
Jul 29 10:28:52.139770 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.139741    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xhv8g"
Jul 29 10:28:52.141678 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.141659    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 10:28:52.141729 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.141673    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 10:28:52.142017 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.141825    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-240.ec2.internal\" not found"
Jul 29 10:28:52.142497 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.142479    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 10:28:52.142497 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.142496    2584 factory.go:55] Registering systemd factory
Jul 29 10:28:52.142497 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.142511    2584 factory.go:223] Registration of the systemd container factory successfully
Jul 29 10:28:52.142676 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.142524    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 10:28:52.142676 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.142479    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 10:28:52.142766 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.142699    2584 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 10:28:52.142766 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.142710    2584 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 10:28:52.142766 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.142746    2584 factory.go:153] Registering CRI-O factory
Jul 29 10:28:52.142766 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.142765    2584 factory.go:223] Registration of the crio container factory successfully
Jul 29 10:28:52.142949 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.142814    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 10:28:52.142949 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.142835    2584 factory.go:103] Registering Raw factory
Jul 29 10:28:52.142949 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.142852    2584 manager.go:1196] Started watching for new ooms in manager
Jul 29 10:28:52.144888 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.144668    2584 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-240.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 10:28:52.145082 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.145050    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 10:28:52.145524 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.145504    2584 manager.go:349] Starting recovery of all containers
Jul 29 10:28:52.147821 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.147793    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xhv8g"
Jul 29 10:28:52.157791 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.157770    2584 manager.go:354] Recovery completed
Jul 29 10:28:52.162348 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.162331    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:52.164349 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.164331    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:52.164421 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.164363    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:52.164421 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.164374    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:52.165003 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.164990    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:52.165003 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.164998    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 10:28:52.165098 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.165008    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 10:28:52.165098 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.165024    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 10:28:52.165656 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.165640    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:52.165711 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.165668    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:52.165711 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.165678    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:52.166296 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.166283    2584 policy_none.go:49] "None policy: Start"
Jul 29 10:28:52.166340 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.166300    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 10:28:52.166340 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.166311    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 10:28:52.167217 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.167206    2584 policy_none.go:47] "Start"
Jul 29 10:28:52.205401 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.205381    2584 manager.go:337] "Starting Device Plugin manager"
Jul 29 10:28:52.205527 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.205424    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 10:28:52.205527 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.205437    2584 server.go:87] "Starting device plugin registration server"
Jul 29 10:28:52.205734 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.205721    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 10:28:52.205894 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.205736    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 10:28:52.205894 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.205869    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 10:28:52.205972 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.205963    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 10:28:52.206026 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.205975    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 10:28:52.206784 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.206756    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 10:28:52.206899 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.206823    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-240.ec2.internal\" not found"
Jul 29 10:28:52.247775 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.247721    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 10:28:52.249383 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.249002    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 10:28:52.249383 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.249040    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 10:28:52.249383 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.249063    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 10:28:52.249383 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.249099    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 10:28:52.251364 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.251340    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:52.307234 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.307184    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:52.308217 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.308199    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:52.308322 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.308232    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:52.308322 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.308243    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:52.308322 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.308269    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.316788 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.316724    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.316788 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.316749    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-240.ec2.internal\": node \"ip-10-0-140-240.ec2.internal\" not found"
Jul 29 10:28:52.337931 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.337906    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-240.ec2.internal\" not found"
Jul 29 10:28:52.350206 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.350185    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal"]
Jul 29 10:28:52.350267 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.350247    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:52.351596 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.351578    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:52.351664 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.351611    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:52.351664 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.351622    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:52.352874 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.352861    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:52.353011 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.352997    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.353055 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.353028    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:52.354569 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.354552    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:52.354653 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.354579    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:52.354653 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.354594    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:52.354653 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.354605    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:52.354653 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.354621    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:52.354653 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.354634    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:52.355934 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.355912    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.356036 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.355942    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:52.356729 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.356713    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:52.356825 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.356764    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:52.356825 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.356786    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:52.385649 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.385626    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-240.ec2.internal\" not found" node="ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.390413 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.390393    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-240.ec2.internal\" not found" node="ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.438757 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.438720    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-240.ec2.internal\" not found"
Jul 29 10:28:52.444331 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.444306    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6690eeea8150ab3c894966b36c641644-config\") pod \"kube-apiserver-proxy-ip-10-0-140-240.ec2.internal\" (UID: \"6690eeea8150ab3c894966b36c641644\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.444477 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.444335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9641d31a7bf8816b31354ae4c19175e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal\" (UID: \"9641d31a7bf8816b31354ae4c19175e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.444477 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.444361    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9641d31a7bf8816b31354ae4c19175e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal\" (UID: \"9641d31a7bf8816b31354ae4c19175e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.539925 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.539883    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-240.ec2.internal\" not found"
Jul 29 10:28:52.544526 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.544499    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9641d31a7bf8816b31354ae4c19175e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal\" (UID: \"9641d31a7bf8816b31354ae4c19175e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.544617 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.544569    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9641d31a7bf8816b31354ae4c19175e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal\" (UID: \"9641d31a7bf8816b31354ae4c19175e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.544683 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.544624    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9641d31a7bf8816b31354ae4c19175e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal\" (UID: \"9641d31a7bf8816b31354ae4c19175e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.544683 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.544644    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9641d31a7bf8816b31354ae4c19175e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal\" (UID: \"9641d31a7bf8816b31354ae4c19175e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.544683 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.544662    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6690eeea8150ab3c894966b36c641644-config\") pod \"kube-apiserver-proxy-ip-10-0-140-240.ec2.internal\" (UID: \"6690eeea8150ab3c894966b36c641644\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.544833 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.544722    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6690eeea8150ab3c894966b36c641644-config\") pod \"kube-apiserver-proxy-ip-10-0-140-240.ec2.internal\" (UID: \"6690eeea8150ab3c894966b36c641644\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.640641 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.640559    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-240.ec2.internal\" not found"
Jul 29 10:28:52.690879 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.690835    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.694039 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:52.694020    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal"
Jul 29 10:28:52.740930 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.740727    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-240.ec2.internal\" not found"
Jul 29 10:28:52.847305 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.847260    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-240.ec2.internal\" not found"
Jul 29 10:28:52.948335 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:52.948254    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-240.ec2.internal\" not found"
Jul 29 10:28:53.048452 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:53.048415    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-240.ec2.internal\" not found"
Jul 29 10:28:53.087769 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:53.087737    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 10:28:53.088188 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:53.087908    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 10:28:53.149323 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:53.149292    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-240.ec2.internal\" not found"
Jul 29 10:28:53.152966 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:53.152932    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 10:23:52 +0000 UTC" deadline="2028-03-31 18:47:27.113391571 +0000 UTC"
Jul 29 10:28:53.152966 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:53.152961    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14672h18m33.960433381s"
Jul 29 10:28:53.215486 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:53.215449    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9641d31a7bf8816b31354ae4c19175e5.slice/crio-ccd334c3b0871a4a81777427a87f5b9913595fa77884bcc8f876f7fcfb263d0f WatchSource:0}: Error finding container ccd334c3b0871a4a81777427a87f5b9913595fa77884bcc8f876f7fcfb263d0f: Status 404 returned error can't find the container with id ccd334c3b0871a4a81777427a87f5b9913595fa77884bcc8f876f7fcfb263d0f
Jul 29 10:28:53.220271 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:53.220250    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:28:53.249492 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:53.249448    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-240.ec2.internal\" not found"
Jul 29 10:28:53.252939 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:53.252894    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal" event={"ID":"9641d31a7bf8816b31354ae4c19175e5","Type":"ContainerStarted","Data":"ccd334c3b0871a4a81777427a87f5b9913595fa77884bcc8f876f7fcfb263d0f"}
Jul 29 10:28:53.265205 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:53.265178    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:53.346405 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:53.346369    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal"
Jul 29 10:28:53.365068 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:53.365042    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 10:28:53.366929 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:53.366914    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal"
Jul 29 10:28:53.378980 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:53.378956    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 10:28:53.385751 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:53.385727    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:53.584411 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:53.584378    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:54.006354 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:54.006320    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6690eeea8150ab3c894966b36c641644.slice/crio-8dd64061c5a4aa6a7da036bfa031d32b249fa8ad319d158a5ab4099b84c282a3 WatchSource:0}: Error finding container 8dd64061c5a4aa6a7da036bfa031d32b249fa8ad319d158a5ab4099b84c282a3: Status 404 returned error can't find the container with id 8dd64061c5a4aa6a7da036bfa031d32b249fa8ad319d158a5ab4099b84c282a3
Jul 29 10:28:54.124192 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.124158    2584 apiserver.go:52] "Watching apiserver"
Jul 29 10:28:54.130750 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.130701    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 10:28:54.131190 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.131165    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-4wb9p","openshift-ovn-kubernetes/ovnkube-node-bwrjz","kube-system/global-pull-secret-syncer-nh6vt","kube-system/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s","openshift-cluster-node-tuning-operator/tuned-nhsdc","openshift-dns/node-resolver-9qhbp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal","openshift-network-diagnostics/network-check-target-fmxtb","openshift-network-operator/iptables-alerter-r6wkn","kube-system/konnectivity-agent-zkcxm","openshift-image-registry/node-ca-b46vj","openshift-multus/multus-8lmdp","openshift-multus/multus-additional-cni-plugins-rpbv6"]
Jul 29 10:28:54.132999 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.132978    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r6wkn"
Jul 29 10:28:54.134673 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.134650    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.135872 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.135848    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:28:54.135985 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.135958    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:28:54.136699 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.136548    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nqlh9\""
Jul 29 10:28:54.136699 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.136562    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 10:28:54.136699 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.136578    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:28:54.136699 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.136564    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 10:28:54.137142 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.137124    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 10:28:54.137395 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.137363    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 10:28:54.137540 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.137517    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 10:28:54.137620 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.137597    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.137620 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.137603    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mvcrt\""
Jul 29 10:28:54.137955 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.137930    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 10:28:54.138560 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.138539    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.139785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.139754    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.140376 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.140359    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dqcsf\""
Jul 29 10:28:54.140452 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.140398    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 10:28:54.140645 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.140618    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 10:28:54.141009 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.140989    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:28:54.141096 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.141061    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:28:54.141244 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.141219    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 10:28:54.141592 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.141552    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 10:28:54.141592 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.141564    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 10:28:54.141864 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.141848    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mjgpl\""
Jul 29 10:28:54.142180 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.142163    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9qhbp"
Jul 29 10:28:54.142371 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.142347    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 10:28:54.142544 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.142502    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rkjxd\""
Jul 29 10:28:54.142644 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.142590    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:28:54.143041 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.142929    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 10:28:54.143845 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.143711    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.147919 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.146092    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:28:54.147919 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.146209    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 10:28:54.147919 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.146233    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:28:54.147919 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.146271    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7xnhw\""
Jul 29 10:28:54.147919 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.146574    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 10:28:54.149059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.149037    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zkcxm"
Jul 29 10:28:54.149152 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.149070    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5p886\""
Jul 29 10:28:54.149211 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.149184    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 10:28:54.149520 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.149498    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 10:28:54.149670 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.149648    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 10:28:54.149770 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.149704    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 10:28:54.149840 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.149799    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 10:28:54.150065 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.150048    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 10:28:54.150496 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.150476    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b46vj"
Jul 29 10:28:54.152323 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.152263    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 10:28:54.153019 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.152997    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z4xxv\""
Jul 29 10:28:54.153419 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.153403    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 10:28:54.153645 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.153619    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 10:28:54.153645 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.153640    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 10:28:54.153831 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.153753    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j6p8b\""
Jul 29 10:28:54.154187 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154168    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 10:28:54.154523 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154506    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/798afeb0-e754-4751-98dd-5dca2a693fe6-cni-binary-copy\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.154616 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154545    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-tuned\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.154616 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154571    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-kubelet\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.154616 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154602    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-run-ovn\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.154822 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154628    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-cni-netd\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.154822 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154651    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/154d9288-1e60-4c44-95cc-b4d0111a48ee-ovnkube-config\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.154822 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154677    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-run-multus-certs\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.154822 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154701    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4km7s\" (UniqueName: \"kubernetes.io/projected/108787f9-47d6-42a1-a5bb-157ff9db98cb-kube-api-access-4km7s\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.154822 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154724    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-var-lib-openvswitch\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.154822 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154778    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa4237ef-2f0b-4010-9430-4caa543ee17e-host-slash\") pod \"iptables-alerter-r6wkn\" (UID: \"aa4237ef-2f0b-4010-9430-4caa543ee17e\") " pod="openshift-network-operator/iptables-alerter-r6wkn"
Jul 29 10:28:54.154822 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-run-k8s-cni-cncf-io\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154828    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/798afeb0-e754-4751-98dd-5dca2a693fe6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154845    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-socket-dir\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154861    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-node-log\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154877    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154901    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-hostroot\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154915    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/798afeb0-e754-4751-98dd-5dca2a693fe6-system-cni-dir\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154937    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/798afeb0-e754-4751-98dd-5dca2a693fe6-os-release\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154961    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.154985    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-slash\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155014    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/154d9288-1e60-4c44-95cc-b4d0111a48ee-ovnkube-script-lib\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155030    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-cnibin\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155052    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/798afeb0-e754-4751-98dd-5dca2a693fe6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155069    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155084    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-etc-selinux\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155099    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-systemd-units\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.155231 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155120    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-log-socket\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155142    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-system-cni-dir\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155171    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-device-dir\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155232    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-run\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155289    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqwnd\" (UniqueName: \"kubernetes.io/projected/aa4237ef-2f0b-4010-9430-4caa543ee17e-kube-api-access-hqwnd\") pod \"iptables-alerter-r6wkn\" (UID: \"aa4237ef-2f0b-4010-9430-4caa543ee17e\") " pod="openshift-network-operator/iptables-alerter-r6wkn"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155332    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx2tj\" (UniqueName: \"kubernetes.io/projected/cd136425-7e77-4564-b811-10f0c9088209-kube-api-access-dx2tj\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155362    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-kubernetes\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155397    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-run-netns\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155423    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dn2l\" (UniqueName: \"kubernetes.io/projected/154d9288-1e60-4c44-95cc-b4d0111a48ee-kube-api-access-9dn2l\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155450    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/798afeb0-e754-4751-98dd-5dca2a693fe6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155472    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfpxs\" (UniqueName: \"kubernetes.io/projected/798afeb0-e754-4751-98dd-5dca2a693fe6-kube-api-access-wfpxs\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155488    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-sysctl-conf\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155502    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-systemd\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155541    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdea5088-94eb-4938-938f-ba83acfaa9d2-dbus\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155571    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-var-lib-kubelet\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155598    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-multus-conf-dir\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.156059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155622    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-sysconfig\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155654    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-lib-modules\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155681    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-run-openvswitch\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155730    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-sys-fs\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155766    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-modprobe-d\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155840    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-host\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155872    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktzbx\" (UniqueName: \"kubernetes.io/projected/605d5f71-2bcc-484f-a8fa-08cb75b43fe2-kube-api-access-ktzbx\") pod \"node-resolver-9qhbp\" (UID: \"605d5f71-2bcc-484f-a8fa-08cb75b43fe2\") " pod="openshift-dns/node-resolver-9qhbp"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155896    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-var-lib-kubelet\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155918    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa4237ef-2f0b-4010-9430-4caa543ee17e-iptables-alerter-script\") pod \"iptables-alerter-r6wkn\" (UID: \"aa4237ef-2f0b-4010-9430-4caa543ee17e\") " pod="openshift-network-operator/iptables-alerter-r6wkn"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155950    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-multus-cni-dir\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155973    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97c7050d-3259-4926-83de-dfa81426daeb-cni-binary-copy\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.155999    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-multus-socket-dir-parent\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156024    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97c7050d-3259-4926-83de-dfa81426daeb-multus-daemon-config\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156067    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-sysctl-d\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156103    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605d5f71-2bcc-484f-a8fa-08cb75b43fe2-tmp-dir\") pod \"node-resolver-9qhbp\" (UID: \"605d5f71-2bcc-484f-a8fa-08cb75b43fe2\") " pod="openshift-dns/node-resolver-9qhbp"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156134    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/154d9288-1e60-4c44-95cc-b4d0111a48ee-ovn-node-metrics-cert\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156173    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-var-lib-cni-bin\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.156927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156212    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/108787f9-47d6-42a1-a5bb-157ff9db98cb-tmp\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156237    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-etc-openvswitch\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156277    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-run-netns\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156313    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-registration-dir\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156382    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-sys\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156418    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/605d5f71-2bcc-484f-a8fa-08cb75b43fe2-hosts-file\") pod \"node-resolver-9qhbp\" (UID: \"605d5f71-2bcc-484f-a8fa-08cb75b43fe2\") " pod="openshift-dns/node-resolver-9qhbp"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156443    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-cni-bin\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156482    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/154d9288-1e60-4c44-95cc-b4d0111a48ee-env-overrides\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156515    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-var-lib-cni-multus\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156557    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-etc-kubernetes\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156584    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqjlj\" (UniqueName: \"kubernetes.io/projected/97c7050d-3259-4926-83de-dfa81426daeb-kube-api-access-dqjlj\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156613    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/798afeb0-e754-4751-98dd-5dca2a693fe6-cnibin\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156639    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zkxc\" (UniqueName: \"kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc\") pod \"network-check-target-fmxtb\" (UID: \"6a230e54-bb1a-4893-abb4-798a229d4d9d\") " pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156707    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-run-systemd\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156753    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-os-release\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156779    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdea5088-94eb-4938-938f-ba83acfaa9d2-kubelet-config\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:28:54.157537 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.156809    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.160413 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.160379    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 10:28:54.180960 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.180933    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sbl8g"
Jul 29 10:28:54.188869 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.188848    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sbl8g"
Jul 29 10:28:54.243584 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.243556    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 10:28:54.257466 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.257391    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-var-lib-cni-multus\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.257466 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.257423    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-etc-kubernetes\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.257466 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.257444    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqjlj\" (UniqueName: \"kubernetes.io/projected/97c7050d-3259-4926-83de-dfa81426daeb-kube-api-access-dqjlj\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.257802 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.257470    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/798afeb0-e754-4751-98dd-5dca2a693fe6-cnibin\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.257802 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.257489    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zkxc\" (UniqueName: \"kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc\") pod \"network-check-target-fmxtb\" (UID: \"6a230e54-bb1a-4893-abb4-798a229d4d9d\") " pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:28:54.257802 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.257517    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/19da85ea-28f6-42ad-afc9-d37d9603cd90-serviceca\") pod \"node-ca-b46vj\" (UID: \"19da85ea-28f6-42ad-afc9-d37d9603cd90\") " pod="openshift-image-registry/node-ca-b46vj"
Jul 29 10:28:54.257802 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.257532    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-var-lib-cni-multus\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.257802 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.257544    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-run-systemd\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.257802 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.257590    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-run-systemd\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.257802 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.257596    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-os-release\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.257802 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.257632    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdea5088-94eb-4938-938f-ba83acfaa9d2-kubelet-config\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:28:54.257802 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.257661    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.257802 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.257665    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-os-release\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.257802 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.257687    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/798afeb0-e754-4751-98dd-5dca2a693fe6-cni-binary-copy\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.257988    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal" event={"ID":"6690eeea8150ab3c894966b36c641644","Type":"ContainerStarted","Data":"8dd64061c5a4aa6a7da036bfa031d32b249fa8ad319d158a5ab4099b84c282a3"}
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258030    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-etc-kubernetes\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258061    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/798afeb0-e754-4751-98dd-5dca2a693fe6-cnibin\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258074    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdea5088-94eb-4938-938f-ba83acfaa9d2-kubelet-config\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258088    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-tuned\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258099    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258119    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19da85ea-28f6-42ad-afc9-d37d9603cd90-host\") pod \"node-ca-b46vj\" (UID: \"19da85ea-28f6-42ad-afc9-d37d9603cd90\") " pod="openshift-image-registry/node-ca-b46vj"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258150    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-kubelet\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258177    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-run-ovn\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258204    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-cni-netd\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258208    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-kubelet\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258228    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/154d9288-1e60-4c44-95cc-b4d0111a48ee-ovnkube-config\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258232    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-run-ovn\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258257    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-run-multus-certs\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258266    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-cni-netd\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258285    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4km7s\" (UniqueName: \"kubernetes.io/projected/108787f9-47d6-42a1-a5bb-157ff9db98cb-kube-api-access-4km7s\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258311    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-var-lib-openvswitch\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.258338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258322    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-run-multus-certs\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258338    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa4237ef-2f0b-4010-9430-4caa543ee17e-host-slash\") pod \"iptables-alerter-r6wkn\" (UID: \"aa4237ef-2f0b-4010-9430-4caa543ee17e\") " pod="openshift-network-operator/iptables-alerter-r6wkn"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258388    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/798afeb0-e754-4751-98dd-5dca2a693fe6-cni-binary-copy\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258389    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-var-lib-openvswitch\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258427    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-run-k8s-cni-cncf-io\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258385    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-run-k8s-cni-cncf-io\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258425    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258475    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/798afeb0-e754-4751-98dd-5dca2a693fe6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258501    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-socket-dir\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258521    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-node-log\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258543    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258551    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa4237ef-2f0b-4010-9430-4caa543ee17e-host-slash\") pod \"iptables-alerter-r6wkn\" (UID: \"aa4237ef-2f0b-4010-9430-4caa543ee17e\") " pod="openshift-network-operator/iptables-alerter-r6wkn"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258563    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-hostroot\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258585    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/798afeb0-e754-4751-98dd-5dca2a693fe6-system-cni-dir\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258605    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/798afeb0-e754-4751-98dd-5dca2a693fe6-os-release\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258618    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258627    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:28:54.259367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258646    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/798afeb0-e754-4751-98dd-5dca2a693fe6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258648    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-slash\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258672    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/798afeb0-e754-4751-98dd-5dca2a693fe6-system-cni-dir\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258682    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-slash\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258606    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-node-log\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258707    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/154d9288-1e60-4c44-95cc-b4d0111a48ee-ovnkube-script-lib\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258736    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-cnibin\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258747    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-socket-dir\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258771    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/798afeb0-e754-4751-98dd-5dca2a693fe6-os-release\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258772    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/798afeb0-e754-4751-98dd-5dca2a693fe6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258800    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-cnibin\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258802    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.258823    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258835    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/154d9288-1e60-4c44-95cc-b4d0111a48ee-ovnkube-config\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258847    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258743    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-hostroot\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258839    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-etc-selinux\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.260253 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.258903    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret podName:cdea5088-94eb-4938-938f-ba83acfaa9d2 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:54.758880248 +0000 UTC m=+2.951907591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret") pod "global-pull-secret-syncer-nh6vt" (UID: "cdea5088-94eb-4938-938f-ba83acfaa9d2") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258927    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0d8deb7-ef6f-4094-bdbe-b01ff9de1d6a-konnectivity-ca\") pod \"konnectivity-agent-zkcxm\" (UID: \"a0d8deb7-ef6f-4094-bdbe-b01ff9de1d6a\") " pod="kube-system/konnectivity-agent-zkcxm"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258958    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-systemd-units\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258964    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-etc-selinux\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.258986    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-log-socket\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259030    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-system-cni-dir\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259054    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-systemd-units\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259057    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-device-dir\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259092    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-log-socket\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259129    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-run\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259157    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqwnd\" (UniqueName: \"kubernetes.io/projected/aa4237ef-2f0b-4010-9430-4caa543ee17e-kube-api-access-hqwnd\") pod \"iptables-alerter-r6wkn\" (UID: \"aa4237ef-2f0b-4010-9430-4caa543ee17e\") " pod="openshift-network-operator/iptables-alerter-r6wkn"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259187    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx2tj\" (UniqueName: \"kubernetes.io/projected/cd136425-7e77-4564-b811-10f0c9088209-kube-api-access-dx2tj\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259215    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-kubernetes\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259243    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/154d9288-1e60-4c44-95cc-b4d0111a48ee-ovnkube-script-lib\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259244    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcjqb\" (UniqueName: \"kubernetes.io/projected/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-kube-api-access-tcjqb\") pod \"network-metrics-daemon-4wb9p\" (UID: \"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29\") " pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259291    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-run-netns\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259335    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-run-netns\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.261202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259330    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/798afeb0-e754-4751-98dd-5dca2a693fe6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259370    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dn2l\" (UniqueName: \"kubernetes.io/projected/154d9288-1e60-4c44-95cc-b4d0111a48ee-kube-api-access-9dn2l\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259400    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-device-dir\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259409    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/798afeb0-e754-4751-98dd-5dca2a693fe6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259463    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-system-cni-dir\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259489    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfpxs\" (UniqueName: \"kubernetes.io/projected/798afeb0-e754-4751-98dd-5dca2a693fe6-kube-api-access-wfpxs\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259525    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-run\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259532    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-kubernetes\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259724    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-sysctl-conf\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259775    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-systemd\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259801    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzdpn\" (UniqueName: \"kubernetes.io/projected/19da85ea-28f6-42ad-afc9-d37d9603cd90-kube-api-access-mzdpn\") pod \"node-ca-b46vj\" (UID: \"19da85ea-28f6-42ad-afc9-d37d9603cd90\") " pod="openshift-image-registry/node-ca-b46vj"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259832    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdea5088-94eb-4938-938f-ba83acfaa9d2-dbus\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259859    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-var-lib-kubelet\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259855    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-systemd\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259877    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-sysctl-conf\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259885    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/798afeb0-e754-4751-98dd-5dca2a693fe6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259884    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-multus-conf-dir\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.262124 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259922    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-multus-conf-dir\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259928    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-sysconfig\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259930    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-var-lib-kubelet\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.259978    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-lib-modules\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260004    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-run-openvswitch\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260038    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-sysconfig\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260044    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdea5088-94eb-4938-938f-ba83acfaa9d2-dbus\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260076    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-sys-fs\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260104    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-modprobe-d\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-host\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260151    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktzbx\" (UniqueName: \"kubernetes.io/projected/605d5f71-2bcc-484f-a8fa-08cb75b43fe2-kube-api-access-ktzbx\") pod \"node-resolver-9qhbp\" (UID: \"605d5f71-2bcc-484f-a8fa-08cb75b43fe2\") " pod="openshift-dns/node-resolver-9qhbp"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260161    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-lib-modules\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260169    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-sys-fs\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260179    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-var-lib-kubelet\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260208    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-modprobe-d\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260214    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa4237ef-2f0b-4010-9430-4caa543ee17e-iptables-alerter-script\") pod \"iptables-alerter-r6wkn\" (UID: \"aa4237ef-2f0b-4010-9430-4caa543ee17e\") " pod="openshift-network-operator/iptables-alerter-r6wkn"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260224    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-var-lib-kubelet\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260218    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-host\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.262979 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260253    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-multus-cni-dir\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260278    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97c7050d-3259-4926-83de-dfa81426daeb-cni-binary-copy\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260303    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-multus-socket-dir-parent\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260328    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97c7050d-3259-4926-83de-dfa81426daeb-multus-daemon-config\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260354    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-sysctl-d\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260444    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-sysctl-d\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260508    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-multus-socket-dir-parent\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260648    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-multus-cni-dir\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260670    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605d5f71-2bcc-484f-a8fa-08cb75b43fe2-tmp-dir\") pod \"node-resolver-9qhbp\" (UID: \"605d5f71-2bcc-484f-a8fa-08cb75b43fe2\") " pod="openshift-dns/node-resolver-9qhbp"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260723    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/154d9288-1e60-4c44-95cc-b4d0111a48ee-ovn-node-metrics-cert\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260753    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-var-lib-cni-bin\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260802    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa4237ef-2f0b-4010-9430-4caa543ee17e-iptables-alerter-script\") pod \"iptables-alerter-r6wkn\" (UID: \"aa4237ef-2f0b-4010-9430-4caa543ee17e\") " pod="openshift-network-operator/iptables-alerter-r6wkn"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260823    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/108787f9-47d6-42a1-a5bb-157ff9db98cb-tmp\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260854    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-etc-openvswitch\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260880    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-run-netns\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260905    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-registration-dir\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260931    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-sys\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260955    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/605d5f71-2bcc-484f-a8fa-08cb75b43fe2-hosts-file\") pod \"node-resolver-9qhbp\" (UID: \"605d5f71-2bcc-484f-a8fa-08cb75b43fe2\") " pod="openshift-dns/node-resolver-9qhbp"
Jul 29 10:28:54.263819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.260983    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs\") pod \"network-metrics-daemon-4wb9p\" (UID: \"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29\") " pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.261023    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97c7050d-3259-4926-83de-dfa81426daeb-multus-daemon-config\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.261078    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97c7050d-3259-4926-83de-dfa81426daeb-cni-binary-copy\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.261084    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-var-lib-cni-bin\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.261028    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0d8deb7-ef6f-4094-bdbe-b01ff9de1d6a-agent-certs\") pod \"konnectivity-agent-zkcxm\" (UID: \"a0d8deb7-ef6f-4094-bdbe-b01ff9de1d6a\") " pod="kube-system/konnectivity-agent-zkcxm"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.261126    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-cni-bin\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.261153    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/154d9288-1e60-4c44-95cc-b4d0111a48ee-env-overrides\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.261153    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd136425-7e77-4564-b811-10f0c9088209-registration-dir\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.261208    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605d5f71-2bcc-484f-a8fa-08cb75b43fe2-tmp-dir\") pod \"node-resolver-9qhbp\" (UID: \"605d5f71-2bcc-484f-a8fa-08cb75b43fe2\") " pod="openshift-dns/node-resolver-9qhbp"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.261228    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-host-cni-bin\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.261261    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97c7050d-3259-4926-83de-dfa81426daeb-host-run-netns\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.261299    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/605d5f71-2bcc-484f-a8fa-08cb75b43fe2-hosts-file\") pod \"node-resolver-9qhbp\" (UID: \"605d5f71-2bcc-484f-a8fa-08cb75b43fe2\") " pod="openshift-dns/node-resolver-9qhbp"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.261334    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/108787f9-47d6-42a1-a5bb-157ff9db98cb-sys\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.261379    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-etc-openvswitch\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.261550    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/154d9288-1e60-4c44-95cc-b4d0111a48ee-env-overrides\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.261615    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/154d9288-1e60-4c44-95cc-b4d0111a48ee-run-openvswitch\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.262910    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/108787f9-47d6-42a1-a5bb-157ff9db98cb-etc-tuned\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.263679    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/154d9288-1e60-4c44-95cc-b4d0111a48ee-ovn-node-metrics-cert\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.264361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.263787    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/108787f9-47d6-42a1-a5bb-157ff9db98cb-tmp\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.269265 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.269241    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:54.269370 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.269275    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:54.269370 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.269353    2584 projected.go:196] Error preparing data for projected volume kube-api-access-8zkxc for pod openshift-network-diagnostics/network-check-target-fmxtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:54.269497 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.269481    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc podName:6a230e54-bb1a-4893-abb4-798a229d4d9d nodeName:}" failed. No retries permitted until 2026-07-29 10:28:54.76944104 +0000 UTC m=+2.962468393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8zkxc" (UniqueName: "kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc") pod "network-check-target-fmxtb" (UID: "6a230e54-bb1a-4893-abb4-798a229d4d9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:54.271384 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.271322    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqjlj\" (UniqueName: \"kubernetes.io/projected/97c7050d-3259-4926-83de-dfa81426daeb-kube-api-access-dqjlj\") pod \"multus-8lmdp\" (UID: \"97c7050d-3259-4926-83de-dfa81426daeb\") " pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.271384 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.271352    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqwnd\" (UniqueName: \"kubernetes.io/projected/aa4237ef-2f0b-4010-9430-4caa543ee17e-kube-api-access-hqwnd\") pod \"iptables-alerter-r6wkn\" (UID: \"aa4237ef-2f0b-4010-9430-4caa543ee17e\") " pod="openshift-network-operator/iptables-alerter-r6wkn"
Jul 29 10:28:54.272343 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.272317    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx2tj\" (UniqueName: \"kubernetes.io/projected/cd136425-7e77-4564-b811-10f0c9088209-kube-api-access-dx2tj\") pod \"aws-ebs-csi-driver-node-lwg5s\" (UID: \"cd136425-7e77-4564-b811-10f0c9088209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.272611 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.272587    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4km7s\" (UniqueName: \"kubernetes.io/projected/108787f9-47d6-42a1-a5bb-157ff9db98cb-kube-api-access-4km7s\") pod \"tuned-nhsdc\" (UID: \"108787f9-47d6-42a1-a5bb-157ff9db98cb\") " pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.273810 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.273791    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfpxs\" (UniqueName: \"kubernetes.io/projected/798afeb0-e754-4751-98dd-5dca2a693fe6-kube-api-access-wfpxs\") pod \"multus-additional-cni-plugins-rpbv6\" (UID: \"798afeb0-e754-4751-98dd-5dca2a693fe6\") " pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.273978 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.273958    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktzbx\" (UniqueName: \"kubernetes.io/projected/605d5f71-2bcc-484f-a8fa-08cb75b43fe2-kube-api-access-ktzbx\") pod \"node-resolver-9qhbp\" (UID: \"605d5f71-2bcc-484f-a8fa-08cb75b43fe2\") " pod="openshift-dns/node-resolver-9qhbp"
Jul 29 10:28:54.274293 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.274272    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dn2l\" (UniqueName: \"kubernetes.io/projected/154d9288-1e60-4c44-95cc-b4d0111a48ee-kube-api-access-9dn2l\") pod \"ovnkube-node-bwrjz\" (UID: \"154d9288-1e60-4c44-95cc-b4d0111a48ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.362401 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.362364    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs\") pod \"network-metrics-daemon-4wb9p\" (UID: \"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29\") " pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:28:54.362401 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.362403    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0d8deb7-ef6f-4094-bdbe-b01ff9de1d6a-agent-certs\") pod \"konnectivity-agent-zkcxm\" (UID: \"a0d8deb7-ef6f-4094-bdbe-b01ff9de1d6a\") " pod="kube-system/konnectivity-agent-zkcxm"
Jul 29 10:28:54.362659 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.362455    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/19da85ea-28f6-42ad-afc9-d37d9603cd90-serviceca\") pod \"node-ca-b46vj\" (UID: \"19da85ea-28f6-42ad-afc9-d37d9603cd90\") " pod="openshift-image-registry/node-ca-b46vj"
Jul 29 10:28:54.362659 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.362480    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19da85ea-28f6-42ad-afc9-d37d9603cd90-host\") pod \"node-ca-b46vj\" (UID: \"19da85ea-28f6-42ad-afc9-d37d9603cd90\") " pod="openshift-image-registry/node-ca-b46vj"
Jul 29 10:28:54.362659 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.362536    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0d8deb7-ef6f-4094-bdbe-b01ff9de1d6a-konnectivity-ca\") pod \"konnectivity-agent-zkcxm\" (UID: \"a0d8deb7-ef6f-4094-bdbe-b01ff9de1d6a\") " pod="kube-system/konnectivity-agent-zkcxm"
Jul 29 10:28:54.362659 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.362547    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:54.362659 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.362568    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcjqb\" (UniqueName: \"kubernetes.io/projected/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-kube-api-access-tcjqb\") pod \"network-metrics-daemon-4wb9p\" (UID: \"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29\") " pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:28:54.362659 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.362591    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19da85ea-28f6-42ad-afc9-d37d9603cd90-host\") pod \"node-ca-b46vj\" (UID: \"19da85ea-28f6-42ad-afc9-d37d9603cd90\") " pod="openshift-image-registry/node-ca-b46vj"
Jul 29 10:28:54.362659 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.362599    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzdpn\" (UniqueName: \"kubernetes.io/projected/19da85ea-28f6-42ad-afc9-d37d9603cd90-kube-api-access-mzdpn\") pod \"node-ca-b46vj\" (UID: \"19da85ea-28f6-42ad-afc9-d37d9603cd90\") " pod="openshift-image-registry/node-ca-b46vj"
Jul 29 10:28:54.362659 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.362633    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs podName:06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:54.862612349 +0000 UTC m=+3.055639692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs") pod "network-metrics-daemon-4wb9p" (UID: "06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:54.363110 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.363063    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/19da85ea-28f6-42ad-afc9-d37d9603cd90-serviceca\") pod \"node-ca-b46vj\" (UID: \"19da85ea-28f6-42ad-afc9-d37d9603cd90\") " pod="openshift-image-registry/node-ca-b46vj"
Jul 29 10:28:54.363372 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.363348    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0d8deb7-ef6f-4094-bdbe-b01ff9de1d6a-konnectivity-ca\") pod \"konnectivity-agent-zkcxm\" (UID: \"a0d8deb7-ef6f-4094-bdbe-b01ff9de1d6a\") " pod="kube-system/konnectivity-agent-zkcxm"
Jul 29 10:28:54.365194 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.365173    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0d8deb7-ef6f-4094-bdbe-b01ff9de1d6a-agent-certs\") pod \"konnectivity-agent-zkcxm\" (UID: \"a0d8deb7-ef6f-4094-bdbe-b01ff9de1d6a\") " pod="kube-system/konnectivity-agent-zkcxm"
Jul 29 10:28:54.375150 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.375126    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzdpn\" (UniqueName: \"kubernetes.io/projected/19da85ea-28f6-42ad-afc9-d37d9603cd90-kube-api-access-mzdpn\") pod \"node-ca-b46vj\" (UID: \"19da85ea-28f6-42ad-afc9-d37d9603cd90\") " pod="openshift-image-registry/node-ca-b46vj"
Jul 29 10:28:54.375529 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.375509    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcjqb\" (UniqueName: \"kubernetes.io/projected/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-kube-api-access-tcjqb\") pod \"network-metrics-daemon-4wb9p\" (UID: \"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29\") " pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:28:54.400314 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.400285    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:54.449382 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.449348    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r6wkn"
Jul 29 10:28:54.455846 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.455822    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8lmdp"
Jul 29 10:28:54.459064 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:54.459043    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4237ef_2f0b_4010_9430_4caa543ee17e.slice/crio-b3fb9ab9746c53fa7d592c42ce6c7177e3552548e2cfa0dd2be1d63724022fdb WatchSource:0}: Error finding container b3fb9ab9746c53fa7d592c42ce6c7177e3552548e2cfa0dd2be1d63724022fdb: Status 404 returned error can't find the container with id b3fb9ab9746c53fa7d592c42ce6c7177e3552548e2cfa0dd2be1d63724022fdb
Jul 29 10:28:54.464505 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:54.464484    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c7050d_3259_4926_83de_dfa81426daeb.slice/crio-7657a914729d40f2aaab0e686b5aa054adece5a50aa979269958506a955af61a WatchSource:0}: Error finding container 7657a914729d40f2aaab0e686b5aa054adece5a50aa979269958506a955af61a: Status 404 returned error can't find the container with id 7657a914729d40f2aaab0e686b5aa054adece5a50aa979269958506a955af61a
Jul 29 10:28:54.465326 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.465306    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rpbv6"
Jul 29 10:28:54.472403 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.472368    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s"
Jul 29 10:28:54.480360 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.480341    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nhsdc"
Jul 29 10:28:54.483887 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:54.483865    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798afeb0_e754_4751_98dd_5dca2a693fe6.slice/crio-5b643acc20e752fa725cedfcb6fd0b6ca3755a467c56731a95579e3940088611 WatchSource:0}: Error finding container 5b643acc20e752fa725cedfcb6fd0b6ca3755a467c56731a95579e3940088611: Status 404 returned error can't find the container with id 5b643acc20e752fa725cedfcb6fd0b6ca3755a467c56731a95579e3940088611
Jul 29 10:28:54.484657 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:54.484536    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd136425_7e77_4564_b811_10f0c9088209.slice/crio-eed9c286d5f9a66b79ffc8d6a571b08321d0b5335a3b71afdd6ea9f04b38c5d5 WatchSource:0}: Error finding container eed9c286d5f9a66b79ffc8d6a571b08321d0b5335a3b71afdd6ea9f04b38c5d5: Status 404 returned error can't find the container with id eed9c286d5f9a66b79ffc8d6a571b08321d0b5335a3b71afdd6ea9f04b38c5d5
Jul 29 10:28:54.490462 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.490315    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9qhbp"
Jul 29 10:28:54.491238 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:54.491019    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108787f9_47d6_42a1_a5bb_157ff9db98cb.slice/crio-34fd32c917d8b0cc44a57da12876fdf0fe86afde499efc94a5881c10d92dc1d1 WatchSource:0}: Error finding container 34fd32c917d8b0cc44a57da12876fdf0fe86afde499efc94a5881c10d92dc1d1: Status 404 returned error can't find the container with id 34fd32c917d8b0cc44a57da12876fdf0fe86afde499efc94a5881c10d92dc1d1
Jul 29 10:28:54.496946 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.496925    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:28:54.498170 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:54.498150    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605d5f71_2bcc_484f_a8fa_08cb75b43fe2.slice/crio-81073ea5f4f77eb1d6159d946c1ed3f4aba42bd2dff2fcb8b15a29cbe32f6c06 WatchSource:0}: Error finding container 81073ea5f4f77eb1d6159d946c1ed3f4aba42bd2dff2fcb8b15a29cbe32f6c06: Status 404 returned error can't find the container with id 81073ea5f4f77eb1d6159d946c1ed3f4aba42bd2dff2fcb8b15a29cbe32f6c06
Jul 29 10:28:54.504605 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.504578    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zkcxm"
Jul 29 10:28:54.506148 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:28:54.506113    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154d9288_1e60_4c44_95cc_b4d0111a48ee.slice/crio-bf709ae2afdb981caa34aa9234e3c4bf5edd0c0699c41bd8b737fee2510d6b97 WatchSource:0}: Error finding container bf709ae2afdb981caa34aa9234e3c4bf5edd0c0699c41bd8b737fee2510d6b97: Status 404 returned error can't find the container with id bf709ae2afdb981caa34aa9234e3c4bf5edd0c0699c41bd8b737fee2510d6b97
Jul 29 10:28:54.511047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.511029    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b46vj"
Jul 29 10:28:54.766420 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.766129    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:28:54.766420 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.766304    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:54.766420 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.766374    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret podName:cdea5088-94eb-4938-938f-ba83acfaa9d2 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:55.766353377 +0000 UTC m=+3.959380720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret") pod "global-pull-secret-syncer-nh6vt" (UID: "cdea5088-94eb-4938-938f-ba83acfaa9d2") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:54.867630 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.866997    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs\") pod \"network-metrics-daemon-4wb9p\" (UID: \"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29\") " pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:28:54.867630 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:54.867049    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zkxc\" (UniqueName: \"kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc\") pod \"network-check-target-fmxtb\" (UID: \"6a230e54-bb1a-4893-abb4-798a229d4d9d\") " pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:28:54.867630 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.867181    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:54.867630 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.867195    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:54.867630 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.867208    2584 projected.go:196] Error preparing data for projected volume kube-api-access-8zkxc for pod openshift-network-diagnostics/network-check-target-fmxtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:54.867630 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.867275    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc podName:6a230e54-bb1a-4893-abb4-798a229d4d9d nodeName:}" failed. No retries permitted until 2026-07-29 10:28:55.86725207 +0000 UTC m=+4.060279416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8zkxc" (UniqueName: "kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc") pod "network-check-target-fmxtb" (UID: "6a230e54-bb1a-4893-abb4-798a229d4d9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:54.867630 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.867399    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:54.867630 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:54.867478    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs podName:06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:55.867457923 +0000 UTC m=+4.060485265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs") pod "network-metrics-daemon-4wb9p" (UID: "06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:55.191856 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.191740    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 10:23:54 +0000 UTC" deadline="2028-04-02 18:02:46.097179083 +0000 UTC"
Jul 29 10:28:55.191856 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.191797    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14719h33m50.905386248s"
Jul 29 10:28:55.250091 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.249455    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:28:55.250091 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:55.249594    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:28:55.268710 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.268672    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s" event={"ID":"cd136425-7e77-4564-b811-10f0c9088209","Type":"ContainerStarted","Data":"eed9c286d5f9a66b79ffc8d6a571b08321d0b5335a3b71afdd6ea9f04b38c5d5"}
Jul 29 10:28:55.278490 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.278455    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpbv6" event={"ID":"798afeb0-e754-4751-98dd-5dca2a693fe6","Type":"ContainerStarted","Data":"5b643acc20e752fa725cedfcb6fd0b6ca3755a467c56731a95579e3940088611"}
Jul 29 10:28:55.290426 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.290140    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8lmdp" event={"ID":"97c7050d-3259-4926-83de-dfa81426daeb","Type":"ContainerStarted","Data":"7657a914729d40f2aaab0e686b5aa054adece5a50aa979269958506a955af61a"}
Jul 29 10:28:55.293686 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.293610    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r6wkn" event={"ID":"aa4237ef-2f0b-4010-9430-4caa543ee17e","Type":"ContainerStarted","Data":"b3fb9ab9746c53fa7d592c42ce6c7177e3552548e2cfa0dd2be1d63724022fdb"}
Jul 29 10:28:55.300520 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.300009    2584 generic.go:356] "Generic (PLEG): container finished" podID="9641d31a7bf8816b31354ae4c19175e5" containerID="cd9f1ae02479f6b4683ff2b6b55220d19501f68bbe5af0d013b5d9102b4445e5" exitCode=0
Jul 29 10:28:55.300520 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.300166    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal" event={"ID":"9641d31a7bf8816b31354ae4c19175e5","Type":"ContainerDied","Data":"cd9f1ae02479f6b4683ff2b6b55220d19501f68bbe5af0d013b5d9102b4445e5"}
Jul 29 10:28:55.301836 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.301811    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b46vj" event={"ID":"19da85ea-28f6-42ad-afc9-d37d9603cd90","Type":"ContainerStarted","Data":"5cdae76499ba50f2d360d5c0662f434875de6ff7d02096e18452d3d2d4395ffd"}
Jul 29 10:28:55.303905 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.303844    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zkcxm" event={"ID":"a0d8deb7-ef6f-4094-bdbe-b01ff9de1d6a","Type":"ContainerStarted","Data":"88e1d2dc0fed9fcf4c48e24f01b9607f51b41dfeaa5a621783f5c958fc412f9f"}
Jul 29 10:28:55.306253 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.306200    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz" event={"ID":"154d9288-1e60-4c44-95cc-b4d0111a48ee","Type":"ContainerStarted","Data":"bf709ae2afdb981caa34aa9234e3c4bf5edd0c0699c41bd8b737fee2510d6b97"}
Jul 29 10:28:55.312409 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.312383    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9qhbp" event={"ID":"605d5f71-2bcc-484f-a8fa-08cb75b43fe2","Type":"ContainerStarted","Data":"81073ea5f4f77eb1d6159d946c1ed3f4aba42bd2dff2fcb8b15a29cbe32f6c06"}
Jul 29 10:28:55.315907 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.315884    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nhsdc" event={"ID":"108787f9-47d6-42a1-a5bb-157ff9db98cb","Type":"ContainerStarted","Data":"34fd32c917d8b0cc44a57da12876fdf0fe86afde499efc94a5881c10d92dc1d1"}
Jul 29 10:28:55.776783 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.776743    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:28:55.777024 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:55.776966    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:55.777024 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:55.777041    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret podName:cdea5088-94eb-4938-938f-ba83acfaa9d2 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:57.777020933 +0000 UTC m=+5.970048287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret") pod "global-pull-secret-syncer-nh6vt" (UID: "cdea5088-94eb-4938-938f-ba83acfaa9d2") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:55.878042 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.877999    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs\") pod \"network-metrics-daemon-4wb9p\" (UID: \"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29\") " pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:28:55.878259 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:55.878062    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zkxc\" (UniqueName: \"kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc\") pod \"network-check-target-fmxtb\" (UID: \"6a230e54-bb1a-4893-abb4-798a229d4d9d\") " pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:28:55.878321 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:55.878263    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:55.878321 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:55.878283    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:55.878321 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:55.878297    2584 projected.go:196] Error preparing data for projected volume kube-api-access-8zkxc for pod openshift-network-diagnostics/network-check-target-fmxtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:55.878551 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:55.878359    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc podName:6a230e54-bb1a-4893-abb4-798a229d4d9d nodeName:}" failed. No retries permitted until 2026-07-29 10:28:57.878341537 +0000 UTC m=+6.071368885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8zkxc" (UniqueName: "kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc") pod "network-check-target-fmxtb" (UID: "6a230e54-bb1a-4893-abb4-798a229d4d9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:55.878888 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:55.878782    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:55.878888 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:55.878836    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs podName:06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:57.878819594 +0000 UTC m=+6.071846926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs") pod "network-metrics-daemon-4wb9p" (UID: "06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:56.193061 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:56.192954    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 10:23:54 +0000 UTC" deadline="2028-02-29 05:24:13.313963134 +0000 UTC"
Jul 29 10:28:56.193061 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:56.192996    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13914h55m17.120972013s"
Jul 29 10:28:56.252930 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:56.252127    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:28:56.252930 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:56.252263    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:28:56.252930 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:56.252715    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:28:56.252930 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:56.252845    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:28:57.250416 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:57.250099    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:28:57.250942 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:57.250572    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:28:57.797381 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:57.797334    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:28:57.797567 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:57.797550    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:57.797629 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:57.797618    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret podName:cdea5088-94eb-4938-938f-ba83acfaa9d2 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:01.797598151 +0000 UTC m=+9.990625489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret") pod "global-pull-secret-syncer-nh6vt" (UID: "cdea5088-94eb-4938-938f-ba83acfaa9d2") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:57.899222 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:57.898303    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs\") pod \"network-metrics-daemon-4wb9p\" (UID: \"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29\") " pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:28:57.899222 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:57.898360    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zkxc\" (UniqueName: \"kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc\") pod \"network-check-target-fmxtb\" (UID: \"6a230e54-bb1a-4893-abb4-798a229d4d9d\") " pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:28:57.899222 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:57.898557    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:57.899222 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:57.898579    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:57.899222 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:57.898594    2584 projected.go:196] Error preparing data for projected volume kube-api-access-8zkxc for pod openshift-network-diagnostics/network-check-target-fmxtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:57.899222 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:57.898654    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc podName:6a230e54-bb1a-4893-abb4-798a229d4d9d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:01.898636371 +0000 UTC m=+10.091663700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8zkxc" (UniqueName: "kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc") pod "network-check-target-fmxtb" (UID: "6a230e54-bb1a-4893-abb4-798a229d4d9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:57.899222 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:57.898751    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:57.899222 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:57.898791    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs podName:06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:01.898778888 +0000 UTC m=+10.091806219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs") pod "network-metrics-daemon-4wb9p" (UID: "06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:58.249571 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:58.249477    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:28:58.251487 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:58.249623    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:28:58.251487 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:58.249843    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:28:58.251487 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:58.249938    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:28:59.249992 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:28:59.249953    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:28:59.250188 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:28:59.250109    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:00.250694 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:00.250042    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:00.250694 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:00.250187    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:29:00.250694 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:00.250048    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:00.250694 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:00.250644    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:29:01.249907 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:01.249874    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:01.250110 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:01.250036    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:01.833459 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:01.833412    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:01.834000 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:01.833678    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:01.834000 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:01.833801    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret podName:cdea5088-94eb-4938-938f-ba83acfaa9d2 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:09.8337793 +0000 UTC m=+18.026806630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret") pod "global-pull-secret-syncer-nh6vt" (UID: "cdea5088-94eb-4938-938f-ba83acfaa9d2") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:01.935148 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:01.934737    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zkxc\" (UniqueName: \"kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc\") pod \"network-check-target-fmxtb\" (UID: \"6a230e54-bb1a-4893-abb4-798a229d4d9d\") " pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:01.935148 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:01.934879    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs\") pod \"network-metrics-daemon-4wb9p\" (UID: \"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29\") " pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:01.935148 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:01.934932    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:01.935148 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:01.934962    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:01.935148 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:01.934975    2584 projected.go:196] Error preparing data for projected volume kube-api-access-8zkxc for pod openshift-network-diagnostics/network-check-target-fmxtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:01.935148 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:01.935050    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:01.935148 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:01.935060    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc podName:6a230e54-bb1a-4893-abb4-798a229d4d9d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:09.935037131 +0000 UTC m=+18.128064488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8zkxc" (UniqueName: "kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc") pod "network-check-target-fmxtb" (UID: "6a230e54-bb1a-4893-abb4-798a229d4d9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:01.935148 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:01.935112    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs podName:06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:09.935092654 +0000 UTC m=+18.128119983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs") pod "network-metrics-daemon-4wb9p" (UID: "06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:02.251372 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:02.251213    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:02.251372 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:02.251345    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:29:02.251873 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:02.251717    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:02.251873 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:02.251857    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:29:03.249771 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:03.249739    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:03.250235 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:03.249855    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:04.250144 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:04.249640    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:04.250144 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:04.249800    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:29:04.250670 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:04.250239    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:04.250670 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:04.250343    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:29:05.249658 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:05.249627    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:05.249848 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:05.249771    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:06.250311 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:06.250282    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:06.250770 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:06.250281    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:06.250770 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:06.250421    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:29:06.250770 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:06.250494    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:29:07.249907 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:07.249870    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:07.250090 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:07.249996    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:08.249913 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:08.249875    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:08.250349 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:08.250013    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:29:08.250349 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:08.250071    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:08.250349 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:08.250190    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:29:09.249607 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:09.249574    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:09.249815 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:09.249734    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:09.899052 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:09.899022    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:09.899487 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:09.899153    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:09.899487 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:09.899223    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret podName:cdea5088-94eb-4938-938f-ba83acfaa9d2 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:25.899204383 +0000 UTC m=+34.092231724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret") pod "global-pull-secret-syncer-nh6vt" (UID: "cdea5088-94eb-4938-938f-ba83acfaa9d2") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:10.000245 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:10.000211    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs\") pod \"network-metrics-daemon-4wb9p\" (UID: \"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29\") " pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:10.000245 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:10.000250    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zkxc\" (UniqueName: \"kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc\") pod \"network-check-target-fmxtb\" (UID: \"6a230e54-bb1a-4893-abb4-798a229d4d9d\") " pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:10.000476 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:10.000368    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:10.000476 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:10.000407    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:10.000476 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:10.000422    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:10.000476 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:10.000431    2584 projected.go:196] Error preparing data for projected volume kube-api-access-8zkxc for pod openshift-network-diagnostics/network-check-target-fmxtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:10.000476 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:10.000436    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs podName:06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:26.000417504 +0000 UTC m=+34.193444855 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs") pod "network-metrics-daemon-4wb9p" (UID: "06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:10.000476 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:10.000467    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc podName:6a230e54-bb1a-4893-abb4-798a229d4d9d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:26.000456697 +0000 UTC m=+34.193484048 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8zkxc" (UniqueName: "kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc") pod "network-check-target-fmxtb" (UID: "6a230e54-bb1a-4893-abb4-798a229d4d9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:10.250029 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:10.249946    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:10.250190 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:10.250090    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:29:10.250190 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:10.250130    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:10.250295 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:10.250250    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:29:11.249271 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:11.249232    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:11.249710 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:11.249370    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:12.250817 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:12.250784    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:12.251252 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:12.250892    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:29:12.251252 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:12.250992    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:12.251252 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:12.251130    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:29:13.250182 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:13.250150    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:13.250359 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:13.250273    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:14.250935 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.250858    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:14.251386 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:14.250995    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:29:14.252636 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.252030    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:14.252636 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:14.252144    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:29:14.358726 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.358505    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpbv6" event={"ID":"798afeb0-e754-4751-98dd-5dca2a693fe6","Type":"ContainerStarted","Data":"679f6e7ecfa682b2cbd1f8395a454f1b78f5dc0ba4cfdaaede4d85b64c5a37ba"}
Jul 29 10:29:14.361702 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.361383    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal" event={"ID":"6690eeea8150ab3c894966b36c641644","Type":"ContainerStarted","Data":"59e5d4c2fa6ecfaa5e3d2d485d311647e8e2d5e166adef0a27a36dd758817bbb"}
Jul 29 10:29:14.363876 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.363698    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8lmdp" event={"ID":"97c7050d-3259-4926-83de-dfa81426daeb","Type":"ContainerStarted","Data":"0c4660a0c9bab989602d56c2bacc909b97080361ba3361ceaf36e350a2c3d2ef"}
Jul 29 10:29:14.366328 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.366303    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal" event={"ID":"9641d31a7bf8816b31354ae4c19175e5","Type":"ContainerStarted","Data":"b9520c4996be2ceb89475d57c9b009bef44dbb083fbfe9657dca7190e21d7336"}
Jul 29 10:29:14.368147 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.368126    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b46vj" event={"ID":"19da85ea-28f6-42ad-afc9-d37d9603cd90","Type":"ContainerStarted","Data":"e64aa7431d099625faa4465542de7c08aa1f1b57487955333102cb72317ab5f7"}
Jul 29 10:29:14.369714 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.369649    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zkcxm" event={"ID":"a0d8deb7-ef6f-4094-bdbe-b01ff9de1d6a","Type":"ContainerStarted","Data":"072a1890b8f49d1e0ba00163e2ef3104454882938f3c8deb16f71ece18c11442"}
Jul 29 10:29:14.372143 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.371810    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz" event={"ID":"154d9288-1e60-4c44-95cc-b4d0111a48ee","Type":"ContainerStarted","Data":"ccc2f1af91b3c9b95df3a524b6b3be76257e2c53513eb663c3f46caa57054657"}
Jul 29 10:29:14.372143 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.372091    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz" event={"ID":"154d9288-1e60-4c44-95cc-b4d0111a48ee","Type":"ContainerStarted","Data":"0be305405ba89382fddca3155e4c468212e08808df0dff8d985a685253d6553f"}
Jul 29 10:29:14.373340 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.373314    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9qhbp" event={"ID":"605d5f71-2bcc-484f-a8fa-08cb75b43fe2","Type":"ContainerStarted","Data":"f90b2542c61b981730440f0572c831248b0bf0f4426b7d0255386be9052bba33"}
Jul 29 10:29:14.374757 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.374737    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nhsdc" event={"ID":"108787f9-47d6-42a1-a5bb-157ff9db98cb","Type":"ContainerStarted","Data":"223dca9d2ad0b8b98d7a0fe0ff63fee4255c2154f2299ba655c65f967d726a12"}
Jul 29 10:29:14.376180 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.376161    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s" event={"ID":"cd136425-7e77-4564-b811-10f0c9088209","Type":"ContainerStarted","Data":"5bbbe68e7c890424a1cedb11903e17281e8a378f209ff71cec2ddccb4f323927"}
Jul 29 10:29:14.401371 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.401287    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-240.ec2.internal" podStartSLOduration=20.40127201 podStartE2EDuration="20.40127201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:14.400956925 +0000 UTC m=+22.593984273" watchObservedRunningTime="2026-07-29 10:29:14.40127201 +0000 UTC m=+22.594299391"
Jul 29 10:29:14.415560 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.415512    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-240.ec2.internal" podStartSLOduration=20.415497537 podStartE2EDuration="20.415497537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:14.415484034 +0000 UTC m=+22.608511371" watchObservedRunningTime="2026-07-29 10:29:14.415497537 +0000 UTC m=+22.608524883"
Jul 29 10:29:14.434184 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.434138    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8lmdp" podStartSLOduration=1.7699533760000001 podStartE2EDuration="21.434122995s" totalImagesPullingTime="19.664169619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:14.433806459 +0000 UTC m=+22.626833807" watchObservedRunningTime="2026-07-29 10:29:14.434122995 +0000 UTC m=+22.627150342"
Jul 29 10:29:14.450476 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.450406    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nhsdc" podStartSLOduration=1.866319673 podStartE2EDuration="21.450385261s" totalImagesPullingTime="19.584065588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:14.450145061 +0000 UTC m=+22.643172405" watchObservedRunningTime="2026-07-29 10:29:14.450385261 +0000 UTC m=+22.643412609"
Jul 29 10:29:14.466313 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.466261    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9qhbp" podStartSLOduration=2.050123543 podStartE2EDuration="21.46624275s" totalImagesPullingTime="19.416119207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:14.465649694 +0000 UTC m=+22.658677043" watchObservedRunningTime="2026-07-29 10:29:14.46624275 +0000 UTC m=+22.659270099"
Jul 29 10:29:14.481669 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.481622    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zkcxm" podStartSLOduration=1.9155691240000001 podStartE2EDuration="21.481605526s" totalImagesPullingTime="19.566036402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:14.480977535 +0000 UTC m=+22.674004884" watchObservedRunningTime="2026-07-29 10:29:14.481605526 +0000 UTC m=+22.674632873"
Jul 29 10:29:14.496348 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:14.496309    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-b46vj" podStartSLOduration=1.9342350069999998 podStartE2EDuration="21.496293768s" totalImagesPullingTime="19.562058761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:14.49571993 +0000 UTC m=+22.688747274" watchObservedRunningTime="2026-07-29 10:29:14.496293768 +0000 UTC m=+22.689321114"
Jul 29 10:29:15.249853 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:15.249824    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:15.250007 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:15.249926    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:15.380603 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:15.380573    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz" event={"ID":"154d9288-1e60-4c44-95cc-b4d0111a48ee","Type":"ContainerStarted","Data":"6d74fa49e4963a045dbd4d31f805732ed00cc8efe525843a1c7d5d947c4712e8"}
Jul 29 10:29:15.381268 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:15.380657    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz" event={"ID":"154d9288-1e60-4c44-95cc-b4d0111a48ee","Type":"ContainerStarted","Data":"844c29946b94e838f4235caccd89801fcaf97c13f153809dd03c42759c641aa4"}
Jul 29 10:29:15.381268 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:15.380716    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz" event={"ID":"154d9288-1e60-4c44-95cc-b4d0111a48ee","Type":"ContainerStarted","Data":"51a4b73797c7ca86b01186e6ea33bcde41a56b176daedc725c0aef02f3723a03"}
Jul 29 10:29:15.381268 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:15.380750    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz" event={"ID":"154d9288-1e60-4c44-95cc-b4d0111a48ee","Type":"ContainerStarted","Data":"5199a9a85c61351c5c257d3f721d610639e034fd91a7410c511721b805001f21"}
Jul 29 10:29:15.381995 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:15.381973    2584 generic.go:356] "Generic (PLEG): container finished" podID="798afeb0-e754-4751-98dd-5dca2a693fe6" containerID="679f6e7ecfa682b2cbd1f8395a454f1b78f5dc0ba4cfdaaede4d85b64c5a37ba" exitCode=0
Jul 29 10:29:15.382111 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:15.382090    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpbv6" event={"ID":"798afeb0-e754-4751-98dd-5dca2a693fe6","Type":"ContainerDied","Data":"679f6e7ecfa682b2cbd1f8395a454f1b78f5dc0ba4cfdaaede4d85b64c5a37ba"}
Jul 29 10:29:15.473396 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:15.473366    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 10:29:16.221123 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:16.220958    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T10:29:15.473390131Z","UUID":"533673e9-7d8a-447e-be14-1f682476c81d","Handler":null,"Name":"","Endpoint":""}
Jul 29 10:29:16.223024 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:16.222997    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 10:29:16.223138 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:16.223040    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 10:29:16.249548 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:16.249514    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:16.249742 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:16.249571    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:16.249742 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:16.249733    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:29:16.250068 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:16.250027    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:29:16.386145 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:16.386109    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s" event={"ID":"cd136425-7e77-4564-b811-10f0c9088209","Type":"ContainerStarted","Data":"9baf95743f96327858218bdcbde746a838b4424fd86dcc17f04501da386f87c8"}
Jul 29 10:29:16.387788 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:16.387758    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r6wkn" event={"ID":"aa4237ef-2f0b-4010-9430-4caa543ee17e","Type":"ContainerStarted","Data":"607a73345db93c0dd698d95053eb6a256844c77ae5ebe344d4cadcc4de5ce753"}
Jul 29 10:29:16.403527 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:16.403471    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r6wkn" podStartSLOduration=8.341578209 podStartE2EDuration="23.403457004s" totalImagesPullingTime="15.061878795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:16.402754211 +0000 UTC m=+24.595781559" watchObservedRunningTime="2026-07-29 10:29:16.403457004 +0000 UTC m=+24.596484358"
Jul 29 10:29:17.250093 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:17.250009    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:17.250241 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:17.250137    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:17.394814 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:17.394774    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz" event={"ID":"154d9288-1e60-4c44-95cc-b4d0111a48ee","Type":"ContainerStarted","Data":"0367187dd56a70f502dfcc5993f27c6efc055c7f4d85a000d31242f4ea368493"}
Jul 29 10:29:17.397233 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:17.397206    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s" event={"ID":"cd136425-7e77-4564-b811-10f0c9088209","Type":"ContainerStarted","Data":"0056de4d8ae2563866bd38eeb38cd3e41a52fac31623d8424ef866b2bfa03bb3"}
Jul 29 10:29:17.415320 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:17.415268    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwg5s" podStartSLOduration=2.788467118 podStartE2EDuration="24.415254533s" totalImagesPullingTime="21.626787415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:17.41510928 +0000 UTC m=+25.608136627" watchObservedRunningTime="2026-07-29 10:29:17.415254533 +0000 UTC m=+25.608281923"
Jul 29 10:29:17.972589 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:17.972557    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zkcxm"
Jul 29 10:29:17.973174 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:17.973151    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zkcxm"
Jul 29 10:29:18.250162 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:18.250089    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:18.250325 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:18.250090    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:18.250325 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:18.250230    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:29:18.250325 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:18.250266    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:29:18.400960 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:18.400928    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zkcxm"
Jul 29 10:29:18.401590 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:18.401541    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zkcxm"
Jul 29 10:29:19.250387 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:19.250153    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:19.250620 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:19.250497    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:20.249821 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:20.249611    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:20.250812 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:20.249668    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:20.250812 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:20.249907    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:29:20.250812 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:20.249958    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:29:20.406837 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:20.406800    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz" event={"ID":"154d9288-1e60-4c44-95cc-b4d0111a48ee","Type":"ContainerStarted","Data":"1ba0dabeb5d5d5f8c470c650a6436fdcb9d583bf8ab5b93defd44e50fe2660a8"}
Jul 29 10:29:20.407046 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:20.407027    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:29:20.408740 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:20.408711    2584 generic.go:356] "Generic (PLEG): container finished" podID="798afeb0-e754-4751-98dd-5dca2a693fe6" containerID="517358488c0d004f1c15051176fe41bab6c606b2ec36f6a5e9a9a04e8ee70201" exitCode=0
Jul 29 10:29:20.408837 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:20.408784    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpbv6" event={"ID":"798afeb0-e754-4751-98dd-5dca2a693fe6","Type":"ContainerDied","Data":"517358488c0d004f1c15051176fe41bab6c606b2ec36f6a5e9a9a04e8ee70201"}
Jul 29 10:29:20.422967 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:20.422899    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:29:20.440383 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:20.440343    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz" podStartSLOduration=7.8243440159999995 podStartE2EDuration="27.440329738s" totalImagesPullingTime="19.615985722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:20.438604246 +0000 UTC m=+28.631631594" watchObservedRunningTime="2026-07-29 10:29:20.440329738 +0000 UTC m=+28.633357086"
Jul 29 10:29:21.249894 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:21.249858    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:21.250406 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:21.250022    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:21.411724 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:21.411690    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:29:21.411898 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:21.411776    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:29:21.428061 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:21.428040    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:29:21.501333 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:21.501267    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fmxtb"]
Jul 29 10:29:21.501442 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:21.501401    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:21.501499 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:21.501482    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:29:21.504096 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:21.504072    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nh6vt"]
Jul 29 10:29:21.504199 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:21.504165    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:21.504293 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:21.504267    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:21.518903 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:21.518843    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4wb9p"]
Jul 29 10:29:21.519022 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:21.518954    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:21.519085 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:21.519060    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:29:22.414815 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:22.414779    2584 generic.go:356] "Generic (PLEG): container finished" podID="798afeb0-e754-4751-98dd-5dca2a693fe6" containerID="c3ec5fecc790a2157236f56d5503f388ab6af94cd2d7f9a65f11be6ad235cedd" exitCode=0
Jul 29 10:29:22.415269 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:22.414860    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpbv6" event={"ID":"798afeb0-e754-4751-98dd-5dca2a693fe6","Type":"ContainerDied","Data":"c3ec5fecc790a2157236f56d5503f388ab6af94cd2d7f9a65f11be6ad235cedd"}
Jul 29 10:29:23.249483 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:23.249453    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:23.249631 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:23.249552    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:23.249631 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:23.249453    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:23.249631 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:23.249453    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:23.249631 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:23.249620    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:29:23.249836 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:23.249726    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:29:24.421630 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:24.421376    2584 generic.go:356] "Generic (PLEG): container finished" podID="798afeb0-e754-4751-98dd-5dca2a693fe6" containerID="4a399de08526622c411f0bb1f86f6592f9a587b52c8fb2c285ab56d89ca5081b" exitCode=0
Jul 29 10:29:24.421630 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:24.421449    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpbv6" event={"ID":"798afeb0-e754-4751-98dd-5dca2a693fe6","Type":"ContainerDied","Data":"4a399de08526622c411f0bb1f86f6592f9a587b52c8fb2c285ab56d89ca5081b"}
Jul 29 10:29:25.249618 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:25.249586    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:25.249618 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:25.249606    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:25.249866 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:25.249716    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:25.249866 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:25.249735    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:25.249866 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:25.249789    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:29:25.249984 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:25.249870    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:29:25.917856 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:25.917816    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:25.918309 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:25.917995    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:25.918309 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:25.918079    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret podName:cdea5088-94eb-4938-938f-ba83acfaa9d2 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:57.918057634 +0000 UTC m=+66.111084969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret") pod "global-pull-secret-syncer-nh6vt" (UID: "cdea5088-94eb-4938-938f-ba83acfaa9d2") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:26.019048 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:26.018939    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs\") pod \"network-metrics-daemon-4wb9p\" (UID: \"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29\") " pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:26.019048 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:26.018992    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zkxc\" (UniqueName: \"kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc\") pod \"network-check-target-fmxtb\" (UID: \"6a230e54-bb1a-4893-abb4-798a229d4d9d\") " pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:26.019342 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:26.019099    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:26.019342 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:26.019145    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:26.019342 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:26.019162    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:26.019342 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:26.019175    2584 projected.go:196] Error preparing data for projected volume kube-api-access-8zkxc for pod openshift-network-diagnostics/network-check-target-fmxtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:26.019342 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:26.019184    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs podName:06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:58.019165604 +0000 UTC m=+66.212192929 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs") pod "network-metrics-daemon-4wb9p" (UID: "06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:26.019342 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:26.019226    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc podName:6a230e54-bb1a-4893-abb4-798a229d4d9d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:58.019211642 +0000 UTC m=+66.212238981 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8zkxc" (UniqueName: "kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc") pod "network-check-target-fmxtb" (UID: "6a230e54-bb1a-4893-abb4-798a229d4d9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:27.249515 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:27.249485    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:27.249976 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:27.249491    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:27.249976 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:27.249590    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4wb9p" podUID="06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29"
Jul 29 10:29:27.249976 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:27.249491    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:27.249976 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:27.249661    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmxtb" podUID="6a230e54-bb1a-4893-abb4-798a229d4d9d"
Jul 29 10:29:27.249976 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:27.249736    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nh6vt" podUID="cdea5088-94eb-4938-938f-ba83acfaa9d2"
Jul 29 10:29:28.178811 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.178783    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-240.ec2.internal" event="NodeReady"
Jul 29 10:29:28.178997 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.178961    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 10:29:28.241188 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.241143    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vgmp2"]
Jul 29 10:29:28.243743 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.243718    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ggdhq"]
Jul 29 10:29:28.243905 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.243858    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:28.245354 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.245336    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ggdhq"
Jul 29 10:29:28.247154 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.247126    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 10:29:28.247410 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.247395    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9rdn4\""
Jul 29 10:29:28.247583 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.247560    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 10:29:28.248025 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.248007    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 10:29:28.248555 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.248535    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 10:29:28.248664 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.248614    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 10:29:28.248664 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.248648    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nxd67\""
Jul 29 10:29:28.255083 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.254932    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vgmp2"]
Jul 29 10:29:28.263374 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.263347    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ggdhq"]
Jul 29 10:29:28.335346 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.335314    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:28.335528 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.335354    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74e2ee97-b7ae-45a8-a487-3061cc9bba14-config-volume\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:28.335528 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.335379    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6z7h\" (UniqueName: \"kubernetes.io/projected/62d848b6-fe3c-489f-8f2d-af1a97db4032-kube-api-access-j6z7h\") pod \"ingress-canary-ggdhq\" (UID: \"62d848b6-fe3c-489f-8f2d-af1a97db4032\") " pod="openshift-ingress-canary/ingress-canary-ggdhq"
Jul 29 10:29:28.335528 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.335499    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62d848b6-fe3c-489f-8f2d-af1a97db4032-cert\") pod \"ingress-canary-ggdhq\" (UID: \"62d848b6-fe3c-489f-8f2d-af1a97db4032\") " pod="openshift-ingress-canary/ingress-canary-ggdhq"
Jul 29 10:29:28.335670 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.335547    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e2ee97-b7ae-45a8-a487-3061cc9bba14-tmp-dir\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:28.335670 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.335578    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94klm\" (UniqueName: \"kubernetes.io/projected/74e2ee97-b7ae-45a8-a487-3061cc9bba14-kube-api-access-94klm\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:28.436999 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.436906    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e2ee97-b7ae-45a8-a487-3061cc9bba14-tmp-dir\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:28.436999 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.436943    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94klm\" (UniqueName: \"kubernetes.io/projected/74e2ee97-b7ae-45a8-a487-3061cc9bba14-kube-api-access-94klm\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:28.437210 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.437001    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:28.437210 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.437035    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74e2ee97-b7ae-45a8-a487-3061cc9bba14-config-volume\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:28.437210 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.437060    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6z7h\" (UniqueName: \"kubernetes.io/projected/62d848b6-fe3c-489f-8f2d-af1a97db4032-kube-api-access-j6z7h\") pod \"ingress-canary-ggdhq\" (UID: \"62d848b6-fe3c-489f-8f2d-af1a97db4032\") " pod="openshift-ingress-canary/ingress-canary-ggdhq"
Jul 29 10:29:28.437210 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.437110    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62d848b6-fe3c-489f-8f2d-af1a97db4032-cert\") pod \"ingress-canary-ggdhq\" (UID: \"62d848b6-fe3c-489f-8f2d-af1a97db4032\") " pod="openshift-ingress-canary/ingress-canary-ggdhq"
Jul 29 10:29:28.437210 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:28.437126    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:28.437210 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:28.437202    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls podName:74e2ee97-b7ae-45a8-a487-3061cc9bba14 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:28.937182471 +0000 UTC m=+37.130209812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls") pod "dns-default-vgmp2" (UID: "74e2ee97-b7ae-45a8-a487-3061cc9bba14") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:28.437490 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.437303    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e2ee97-b7ae-45a8-a487-3061cc9bba14-tmp-dir\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:28.437645 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.437616    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74e2ee97-b7ae-45a8-a487-3061cc9bba14-config-volume\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:28.441930 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.441910    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62d848b6-fe3c-489f-8f2d-af1a97db4032-cert\") pod \"ingress-canary-ggdhq\" (UID: \"62d848b6-fe3c-489f-8f2d-af1a97db4032\") " pod="openshift-ingress-canary/ingress-canary-ggdhq"
Jul 29 10:29:28.447202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.447179    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94klm\" (UniqueName: \"kubernetes.io/projected/74e2ee97-b7ae-45a8-a487-3061cc9bba14-kube-api-access-94klm\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:28.447464 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.447444    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6z7h\" (UniqueName: \"kubernetes.io/projected/62d848b6-fe3c-489f-8f2d-af1a97db4032-kube-api-access-j6z7h\") pod \"ingress-canary-ggdhq\" (UID: \"62d848b6-fe3c-489f-8f2d-af1a97db4032\") " pod="openshift-ingress-canary/ingress-canary-ggdhq"
Jul 29 10:29:28.566365 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.566330    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ggdhq"
Jul 29 10:29:28.855211 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.855062    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ggdhq"]
Jul 29 10:29:28.858560 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:29:28.858537    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d848b6_fe3c_489f_8f2d_af1a97db4032.slice/crio-69b5118b7f4e2060c746f3255ccecd8cff3066db566035b26c38c41859e45021 WatchSource:0}: Error finding container 69b5118b7f4e2060c746f3255ccecd8cff3066db566035b26c38c41859e45021: Status 404 returned error can't find the container with id 69b5118b7f4e2060c746f3255ccecd8cff3066db566035b26c38c41859e45021
Jul 29 10:29:28.941080 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:28.941052    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:28.941198 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:28.941182    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:28.941344 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:28.941251    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls podName:74e2ee97-b7ae-45a8-a487-3061cc9bba14 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:29.941234029 +0000 UTC m=+38.134261355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls") pod "dns-default-vgmp2" (UID: "74e2ee97-b7ae-45a8-a487-3061cc9bba14") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:29.249398 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:29.249369    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:29.249398 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:29.249395    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:29.249652 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:29.249395    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:29.252787 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:29.252765    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 10:29:29.252916 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:29.252807    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 10:29:29.252916 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:29.252821    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 10:29:29.254042 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:29.254021    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 10:29:29.254126 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:29.254091    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h2t8s\""
Jul 29 10:29:29.254126 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:29.254030    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kqgtv\""
Jul 29 10:29:29.437306 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:29.437269    2584 generic.go:356] "Generic (PLEG): container finished" podID="798afeb0-e754-4751-98dd-5dca2a693fe6" containerID="154d7592ef2f679c4f91363d799dc342adde73604627941db36ed7affabf6f8e" exitCode=0
Jul 29 10:29:29.437774 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:29.437366    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpbv6" event={"ID":"798afeb0-e754-4751-98dd-5dca2a693fe6","Type":"ContainerDied","Data":"154d7592ef2f679c4f91363d799dc342adde73604627941db36ed7affabf6f8e"}
Jul 29 10:29:29.438790 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:29.438763    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ggdhq" event={"ID":"62d848b6-fe3c-489f-8f2d-af1a97db4032","Type":"ContainerStarted","Data":"69b5118b7f4e2060c746f3255ccecd8cff3066db566035b26c38c41859e45021"}
Jul 29 10:29:29.949070 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:29.949030    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:29.949255 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:29.949191    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:29.949308 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:29.949264    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls podName:74e2ee97-b7ae-45a8-a487-3061cc9bba14 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:31.949243679 +0000 UTC m=+40.142271008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls") pod "dns-default-vgmp2" (UID: "74e2ee97-b7ae-45a8-a487-3061cc9bba14") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:30.445055 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:30.445013    2584 generic.go:356] "Generic (PLEG): container finished" podID="798afeb0-e754-4751-98dd-5dca2a693fe6" containerID="5b6834404816bec5b90fd3cc68825e08b2b9ef86527ae2eb30ed3af95358c9e8" exitCode=0
Jul 29 10:29:30.445519 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:30.445067    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpbv6" event={"ID":"798afeb0-e754-4751-98dd-5dca2a693fe6","Type":"ContainerDied","Data":"5b6834404816bec5b90fd3cc68825e08b2b9ef86527ae2eb30ed3af95358c9e8"}
Jul 29 10:29:31.450959 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:31.450718    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpbv6" event={"ID":"798afeb0-e754-4751-98dd-5dca2a693fe6","Type":"ContainerStarted","Data":"fc09875155ae8894e1b22223238abd279c962491fb28928d1a51222a8b03631a"}
Jul 29 10:29:31.452128 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:31.452109    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ggdhq" event={"ID":"62d848b6-fe3c-489f-8f2d-af1a97db4032","Type":"ContainerStarted","Data":"84bd10bfdc7fdf0e66e768139717c73a43156133718eb570d511ca72a31f2a9d"}
Jul 29 10:29:31.475535 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:31.475479    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rpbv6" podStartSLOduration=8.144737908 podStartE2EDuration="38.475461565s" totalImagesPullingTime="30.330723657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:31.475174846 +0000 UTC m=+39.668202196" watchObservedRunningTime="2026-07-29 10:29:31.475461565 +0000 UTC m=+39.668488912"
Jul 29 10:29:31.490449 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:31.490407    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ggdhq" podStartSLOduration=1.491204639 podStartE2EDuration="3.490392729s" totalImagesPullingTime="1.99918809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:31.490085992 +0000 UTC m=+39.683113350" watchObservedRunningTime="2026-07-29 10:29:31.490392729 +0000 UTC m=+39.683420075"
Jul 29 10:29:31.963196 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:31.963157    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:31.963364 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:31.963263    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:31.963364 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:31.963315    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls podName:74e2ee97-b7ae-45a8-a487-3061cc9bba14 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:35.963302101 +0000 UTC m=+44.156329426 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls") pod "dns-default-vgmp2" (UID: "74e2ee97-b7ae-45a8-a487-3061cc9bba14") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:32.538740 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.538704    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bj476"]
Jul 29 10:29:32.574432 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.574402    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bj476"]
Jul 29 10:29:32.574565 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.574487    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bj476"
Jul 29 10:29:32.577493 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.577467    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 29 10:29:32.577630 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.577587    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:29:32.578976 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.578957    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7qkmn\""
Jul 29 10:29:32.642797 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.642766    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx"]
Jul 29 10:29:32.666937 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.666903    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx"]
Jul 29 10:29:32.667066 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.666999    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx"
Jul 29 10:29:32.669461 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.669433    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mdh2\" (UniqueName: \"kubernetes.io/projected/7e6e3db5-2509-478a-989c-f26b47bf1f76-kube-api-access-7mdh2\") pod \"volume-data-source-validator-549c6d54cf-bj476\" (UID: \"7e6e3db5-2509-478a-989c-f26b47bf1f76\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bj476"
Jul 29 10:29:32.669935 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.669913    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 29 10:29:32.669935 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.669925    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gdrpj\""
Jul 29 10:29:32.670108 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.669948    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 29 10:29:32.670169 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.670118    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:29:32.670216 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.670182    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 29 10:29:32.754402 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.754370    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85b7597dd7-9qbmm"]
Jul 29 10:29:32.769929 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.769902    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg4l2\" (UniqueName: \"kubernetes.io/projected/8fa58554-9c9a-445a-8e67-b6a0425565f5-kube-api-access-dg4l2\") pod \"service-ca-operator-5d6c75f84f-b7nxx\" (UID: \"8fa58554-9c9a-445a-8e67-b6a0425565f5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx"
Jul 29 10:29:32.770037 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.769951    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mdh2\" (UniqueName: \"kubernetes.io/projected/7e6e3db5-2509-478a-989c-f26b47bf1f76-kube-api-access-7mdh2\") pod \"volume-data-source-validator-549c6d54cf-bj476\" (UID: \"7e6e3db5-2509-478a-989c-f26b47bf1f76\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bj476"
Jul 29 10:29:32.770037 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.769976    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fa58554-9c9a-445a-8e67-b6a0425565f5-serving-cert\") pod \"service-ca-operator-5d6c75f84f-b7nxx\" (UID: \"8fa58554-9c9a-445a-8e67-b6a0425565f5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx"
Jul 29 10:29:32.770037 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.769993    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8fa58554-9c9a-445a-8e67-b6a0425565f5-config\") pod \"service-ca-operator-5d6c75f84f-b7nxx\" (UID: \"8fa58554-9c9a-445a-8e67-b6a0425565f5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx"
Jul 29 10:29:32.771649 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.771634    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.774712 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.774696    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8g6k9\""
Jul 29 10:29:32.774886 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.774873    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 10:29:32.775115 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.775101    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 10:29:32.775154 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.775103    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 10:29:32.778837 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.778816    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85b7597dd7-9qbmm"]
Jul 29 10:29:32.783339 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.783319    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 10:29:32.785983 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.785961    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mdh2\" (UniqueName: \"kubernetes.io/projected/7e6e3db5-2509-478a-989c-f26b47bf1f76-kube-api-access-7mdh2\") pod \"volume-data-source-validator-549c6d54cf-bj476\" (UID: \"7e6e3db5-2509-478a-989c-f26b47bf1f76\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bj476"
Jul 29 10:29:32.870407 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.870320    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/631e8b8d-9ada-4a4b-b163-4d7c83311599-installation-pull-secrets\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.870407 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.870360    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/631e8b8d-9ada-4a4b-b163-4d7c83311599-trusted-ca\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.870407 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.870378    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2l7l\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-kube-api-access-t2l7l\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.870718 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.870457    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg4l2\" (UniqueName: \"kubernetes.io/projected/8fa58554-9c9a-445a-8e67-b6a0425565f5-kube-api-access-dg4l2\") pod \"service-ca-operator-5d6c75f84f-b7nxx\" (UID: \"8fa58554-9c9a-445a-8e67-b6a0425565f5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx"
Jul 29 10:29:32.870718 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.870481    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/631e8b8d-9ada-4a4b-b163-4d7c83311599-image-registry-private-configuration\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.870718 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.870499    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-bound-sa-token\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.870718 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.870528    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/631e8b8d-9ada-4a4b-b163-4d7c83311599-ca-trust-extracted\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.870718 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.870628    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fa58554-9c9a-445a-8e67-b6a0425565f5-serving-cert\") pod \"service-ca-operator-5d6c75f84f-b7nxx\" (UID: \"8fa58554-9c9a-445a-8e67-b6a0425565f5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx"
Jul 29 10:29:32.870718 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.870668    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8fa58554-9c9a-445a-8e67-b6a0425565f5-config\") pod \"service-ca-operator-5d6c75f84f-b7nxx\" (UID: \"8fa58554-9c9a-445a-8e67-b6a0425565f5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx"
Jul 29 10:29:32.870940 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.870724    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.870940 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.870750    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-certificates\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.871219 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.871198    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8fa58554-9c9a-445a-8e67-b6a0425565f5-config\") pod \"service-ca-operator-5d6c75f84f-b7nxx\" (UID: \"8fa58554-9c9a-445a-8e67-b6a0425565f5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx"
Jul 29 10:29:32.872763 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.872748    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fa58554-9c9a-445a-8e67-b6a0425565f5-serving-cert\") pod \"service-ca-operator-5d6c75f84f-b7nxx\" (UID: \"8fa58554-9c9a-445a-8e67-b6a0425565f5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx"
Jul 29 10:29:32.879915 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.879885    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg4l2\" (UniqueName: \"kubernetes.io/projected/8fa58554-9c9a-445a-8e67-b6a0425565f5-kube-api-access-dg4l2\") pod \"service-ca-operator-5d6c75f84f-b7nxx\" (UID: \"8fa58554-9c9a-445a-8e67-b6a0425565f5\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx"
Jul 29 10:29:32.882758 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.882736    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bj476"
Jul 29 10:29:32.972164 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.972130    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.972307 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.972181    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-certificates\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.972307 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.972209    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/631e8b8d-9ada-4a4b-b163-4d7c83311599-installation-pull-secrets\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.972307 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.972240    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/631e8b8d-9ada-4a4b-b163-4d7c83311599-trusted-ca\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.972307 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.972266    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2l7l\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-kube-api-access-t2l7l\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.972307 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:32.972278    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:32.972307 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:32.972300    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b7597dd7-9qbmm: secret "image-registry-tls" not found
Jul 29 10:29:32.972610 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.972340    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/631e8b8d-9ada-4a4b-b163-4d7c83311599-image-registry-private-configuration\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.972610 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:32.972367    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls podName:631e8b8d-9ada-4a4b-b163-4d7c83311599 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:33.472345487 +0000 UTC m=+41.665372829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls") pod "image-registry-85b7597dd7-9qbmm" (UID: "631e8b8d-9ada-4a4b-b163-4d7c83311599") : secret "image-registry-tls" not found
Jul 29 10:29:32.972610 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.972403    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-bound-sa-token\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.972610 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.972444    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/631e8b8d-9ada-4a4b-b163-4d7c83311599-ca-trust-extracted\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.973162 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.972876    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-certificates\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.973272 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.973185    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/631e8b8d-9ada-4a4b-b163-4d7c83311599-ca-trust-extracted\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.973511 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.973491    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/631e8b8d-9ada-4a4b-b163-4d7c83311599-trusted-ca\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.975614 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.975594    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/631e8b8d-9ada-4a4b-b163-4d7c83311599-image-registry-private-configuration\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.975747 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.975726    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/631e8b8d-9ada-4a4b-b163-4d7c83311599-installation-pull-secrets\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.981377 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.981358    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2l7l\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-kube-api-access-t2l7l\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.981466 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.981450    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-bound-sa-token\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:32.990362 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:32.990344    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx"
Jul 29 10:29:33.011368 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:33.011346    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bj476"]
Jul 29 10:29:33.014169 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:29:33.014147    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6e3db5_2509_478a_989c_f26b47bf1f76.slice/crio-ff3d8aac6289adaed8a9cc0e7facd800b65bab59c60671b3184e105d423f56c3 WatchSource:0}: Error finding container ff3d8aac6289adaed8a9cc0e7facd800b65bab59c60671b3184e105d423f56c3: Status 404 returned error can't find the container with id ff3d8aac6289adaed8a9cc0e7facd800b65bab59c60671b3184e105d423f56c3
Jul 29 10:29:33.137923 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:33.137852    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx"]
Jul 29 10:29:33.140670 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:29:33.140645    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa58554_9c9a_445a_8e67_b6a0425565f5.slice/crio-5c3c5494da480e2bef96d083f7a1f25452eb2a9c29ce16753bc0e5a1f3cd18b5 WatchSource:0}: Error finding container 5c3c5494da480e2bef96d083f7a1f25452eb2a9c29ce16753bc0e5a1f3cd18b5: Status 404 returned error can't find the container with id 5c3c5494da480e2bef96d083f7a1f25452eb2a9c29ce16753bc0e5a1f3cd18b5
Jul 29 10:29:33.458051 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:33.457967    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx" event={"ID":"8fa58554-9c9a-445a-8e67-b6a0425565f5","Type":"ContainerStarted","Data":"5c3c5494da480e2bef96d083f7a1f25452eb2a9c29ce16753bc0e5a1f3cd18b5"}
Jul 29 10:29:33.459059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:33.459025    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bj476" event={"ID":"7e6e3db5-2509-478a-989c-f26b47bf1f76","Type":"ContainerStarted","Data":"ff3d8aac6289adaed8a9cc0e7facd800b65bab59c60671b3184e105d423f56c3"}
Jul 29 10:29:33.475615 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:33.475593    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:33.475781 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:33.475764    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:33.475844 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:33.475785    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b7597dd7-9qbmm: secret "image-registry-tls" not found
Jul 29 10:29:33.475890 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:33.475850    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls podName:631e8b8d-9ada-4a4b-b163-4d7c83311599 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:34.475829033 +0000 UTC m=+42.668856360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls") pod "image-registry-85b7597dd7-9qbmm" (UID: "631e8b8d-9ada-4a4b-b163-4d7c83311599") : secret "image-registry-tls" not found
Jul 29 10:29:34.485234 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:34.485199    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:34.485706 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:34.485381    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:34.485706 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:34.485402    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b7597dd7-9qbmm: secret "image-registry-tls" not found
Jul 29 10:29:34.485706 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:34.485472    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls podName:631e8b8d-9ada-4a4b-b163-4d7c83311599 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:36.485453052 +0000 UTC m=+44.678480399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls") pod "image-registry-85b7597dd7-9qbmm" (UID: "631e8b8d-9ada-4a4b-b163-4d7c83311599") : secret "image-registry-tls" not found
Jul 29 10:29:35.465372 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:35.465337    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bj476" event={"ID":"7e6e3db5-2509-478a-989c-f26b47bf1f76","Type":"ContainerStarted","Data":"cdd5a313ae1db0c2d0a08efc11532934720a2059c798441f74f2094ff09dcb82"}
Jul 29 10:29:35.482234 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:35.482194    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bj476" podStartSLOduration=1.215071104 podStartE2EDuration="3.4821802s" totalImagesPullingTime="2.267109096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:35.480706268 +0000 UTC m=+43.673733617" watchObservedRunningTime="2026-07-29 10:29:35.4821802 +0000 UTC m=+43.675207573"
Jul 29 10:29:35.996082 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:35.995987    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:35.996533 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:35.996111    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:35.996533 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:35.996166    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls podName:74e2ee97-b7ae-45a8-a487-3061cc9bba14 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:43.996149752 +0000 UTC m=+52.189177077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls") pod "dns-default-vgmp2" (UID: "74e2ee97-b7ae-45a8-a487-3061cc9bba14") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:36.469207 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.469165    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx" event={"ID":"8fa58554-9c9a-445a-8e67-b6a0425565f5","Type":"ContainerStarted","Data":"b9c756ebd1b5a7884f61fc309fadb86a2764819d33a4046f99b5c6d7ede0cb60"}
Jul 29 10:29:36.487271 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.487225    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx" podStartSLOduration=2.024660405 podStartE2EDuration="4.487210039s" totalImagesPullingTime="2.462549634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:36.485403241 +0000 UTC m=+44.678430587" watchObservedRunningTime="2026-07-29 10:29:36.487210039 +0000 UTC m=+44.680237385"
Jul 29 10:29:36.500996 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.500972    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:36.501114 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:36.501099    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:36.501114 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:36.501113    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b7597dd7-9qbmm: secret "image-registry-tls" not found
Jul 29 10:29:36.501217 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:36.501163    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls podName:631e8b8d-9ada-4a4b-b163-4d7c83311599 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:40.501145131 +0000 UTC m=+48.694172480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls") pod "image-registry-85b7597dd7-9qbmm" (UID: "631e8b8d-9ada-4a4b-b163-4d7c83311599") : secret "image-registry-tls" not found
Jul 29 10:29:36.861634 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.861598    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hbvl8"]
Jul 29 10:29:36.874465 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.874439    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:36.877721 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.877692    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l6k5g\""
Jul 29 10:29:36.877845 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.877813    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 10:29:36.878907 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.878888    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 10:29:36.878907 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.878904    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 10:29:36.879088 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.878917    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 10:29:36.892754 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.892735    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hbvl8"]
Jul 29 10:29:36.905118 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.905097    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-crio-socket\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:36.905212 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.905126    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4czd\" (UniqueName: \"kubernetes.io/projected/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-api-access-b4czd\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:36.905212 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.905155    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:36.905312 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.905234    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-data-volume\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:36.905312 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:36.905263    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:37.005764 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:37.005729    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-crio-socket\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:37.006263 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:37.005775    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4czd\" (UniqueName: \"kubernetes.io/projected/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-api-access-b4czd\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:37.006263 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:37.005925    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:37.006263 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:37.005947    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-crio-socket\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:37.006263 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:37.006000    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-data-volume\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:37.006263 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:37.006008    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:37.006263 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:37.006028    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:37.006263 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:37.006064    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm podName:617ed549-8c3d-45f4-8108-19aaf5d0fa2d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:37.506049076 +0000 UTC m=+45.699076430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hbvl8" (UID: "617ed549-8c3d-45f4-8108-19aaf5d0fa2d") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:37.006263 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:37.006120    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:37.006263 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:37.006172    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls podName:617ed549-8c3d-45f4-8108-19aaf5d0fa2d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:37.506158089 +0000 UTC m=+45.699185417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbvl8" (UID: "617ed549-8c3d-45f4-8108-19aaf5d0fa2d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:37.006580 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:37.006315    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-data-volume\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:37.014882 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:37.014852    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4czd\" (UniqueName: \"kubernetes.io/projected/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-api-access-b4czd\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:37.510010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:37.509981    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:37.510202 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:37.510018    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:37.510202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:37.510054    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:37.510202 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:37.510064    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm podName:617ed549-8c3d-45f4-8108-19aaf5d0fa2d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:38.510051075 +0000 UTC m=+46.703078399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hbvl8" (UID: "617ed549-8c3d-45f4-8108-19aaf5d0fa2d") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:37.510202 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:37.510103    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:37.510202 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:37.510146    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls podName:617ed549-8c3d-45f4-8108-19aaf5d0fa2d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:38.510133036 +0000 UTC m=+46.703160365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbvl8" (UID: "617ed549-8c3d-45f4-8108-19aaf5d0fa2d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:38.515994 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:38.515963    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:38.516462 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:38.516070    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:38.516462 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:38.516135    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:38.516462 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:38.516164    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:38.516462 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:38.516213    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls podName:617ed549-8c3d-45f4-8108-19aaf5d0fa2d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:40.516194583 +0000 UTC m=+48.709221912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbvl8" (UID: "617ed549-8c3d-45f4-8108-19aaf5d0fa2d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:38.516462 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:38.516228    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm podName:617ed549-8c3d-45f4-8108-19aaf5d0fa2d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:40.516220955 +0000 UTC m=+48.709248280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hbvl8" (UID: "617ed549-8c3d-45f4-8108-19aaf5d0fa2d") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:38.841443 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:38.841415    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9qhbp_605d5f71-2bcc-484f-a8fa-08cb75b43fe2/dns-node-resolver/0.log"
Jul 29 10:29:39.249504 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:39.249423    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-b779r"]
Jul 29 10:29:39.274553 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:39.274521    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-b779r"]
Jul 29 10:29:39.274715 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:39.274613    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b779r"
Jul 29 10:29:39.277581 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:39.277556    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 29 10:29:39.277581 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:39.277558    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 29 10:29:39.278616 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:39.278594    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2p4nv\""
Jul 29 10:29:39.323115 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:39.323083    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmtj4\" (UniqueName: \"kubernetes.io/projected/f51faa87-2bae-442b-8d9e-01b4d270f0ec-kube-api-access-tmtj4\") pod \"migrator-6865c9967-b779r\" (UID: \"f51faa87-2bae-442b-8d9e-01b4d270f0ec\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b779r"
Jul 29 10:29:39.424210 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:39.424170    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmtj4\" (UniqueName: \"kubernetes.io/projected/f51faa87-2bae-442b-8d9e-01b4d270f0ec-kube-api-access-tmtj4\") pod \"migrator-6865c9967-b779r\" (UID: \"f51faa87-2bae-442b-8d9e-01b4d270f0ec\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b779r"
Jul 29 10:29:39.438001 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:39.437968    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmtj4\" (UniqueName: \"kubernetes.io/projected/f51faa87-2bae-442b-8d9e-01b4d270f0ec-kube-api-access-tmtj4\") pod \"migrator-6865c9967-b779r\" (UID: \"f51faa87-2bae-442b-8d9e-01b4d270f0ec\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b779r"
Jul 29 10:29:39.583806 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:39.583773    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b779r"
Jul 29 10:29:39.709259 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:39.709230    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-b779r"]
Jul 29 10:29:39.712124 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:29:39.712098    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51faa87_2bae_442b_8d9e_01b4d270f0ec.slice/crio-ba5628f7c290c126ba0bdccc436b722cb5e991715e437bea99da695c922c7d28 WatchSource:0}: Error finding container ba5628f7c290c126ba0bdccc436b722cb5e991715e437bea99da695c922c7d28: Status 404 returned error can't find the container with id ba5628f7c290c126ba0bdccc436b722cb5e991715e437bea99da695c922c7d28
Jul 29 10:29:40.041236 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:40.041212    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b46vj_19da85ea-28f6-42ad-afc9-d37d9603cd90/node-ca/0.log"
Jul 29 10:29:40.479125 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:40.479049    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b779r" event={"ID":"f51faa87-2bae-442b-8d9e-01b4d270f0ec","Type":"ContainerStarted","Data":"ba5628f7c290c126ba0bdccc436b722cb5e991715e437bea99da695c922c7d28"}
Jul 29 10:29:40.533735 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:40.533706    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:40.533892 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:40.533771    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:40.533892 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:40.533821    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:40.533892 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:40.533858    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:40.533892 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:40.533881    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b7597dd7-9qbmm: secret "image-registry-tls" not found
Jul 29 10:29:40.533892 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:40.533891    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:40.534134 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:40.533937    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:40.534134 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:40.533959    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls podName:631e8b8d-9ada-4a4b-b163-4d7c83311599 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:48.533940126 +0000 UTC m=+56.726967465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls") pod "image-registry-85b7597dd7-9qbmm" (UID: "631e8b8d-9ada-4a4b-b163-4d7c83311599") : secret "image-registry-tls" not found
Jul 29 10:29:40.534134 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:40.533980    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm podName:617ed549-8c3d-45f4-8108-19aaf5d0fa2d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:44.53397095 +0000 UTC m=+52.726998280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hbvl8" (UID: "617ed549-8c3d-45f4-8108-19aaf5d0fa2d") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:40.534134 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:40.534000    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls podName:617ed549-8c3d-45f4-8108-19aaf5d0fa2d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:44.533988309 +0000 UTC m=+52.727015635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbvl8" (UID: "617ed549-8c3d-45f4-8108-19aaf5d0fa2d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:40.840911 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:40.840880    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ggdhq_62d848b6-fe3c-489f-8f2d-af1a97db4032/serve-healthcheck-canary/0.log"
Jul 29 10:29:41.483104 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:41.483077    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b779r" event={"ID":"f51faa87-2bae-442b-8d9e-01b4d270f0ec","Type":"ContainerStarted","Data":"6db5f53c196dd7c71e1653a397f935f93e188d9c013092c41771164c58d418d2"}
Jul 29 10:29:42.486747 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:42.486716    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b779r" event={"ID":"f51faa87-2bae-442b-8d9e-01b4d270f0ec","Type":"ContainerStarted","Data":"4622fbe455d3e91f11f17458b38d6327fce2955104b04303a4e9cb5b6c8ca325"}
Jul 29 10:29:42.503163 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:42.503115    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b779r" podStartSLOduration=1.888527404 podStartE2EDuration="3.503100976s" totalImagesPullingTime="1.614573572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:42.502448845 +0000 UTC m=+50.695476192" watchObservedRunningTime="2026-07-29 10:29:42.503100976 +0000 UTC m=+50.696128323"
Jul 29 10:29:44.061267 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:44.061236    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:29:44.061737 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:44.061388    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:44.061737 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:44.061448    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls podName:74e2ee97-b7ae-45a8-a487-3061cc9bba14 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:00.061433572 +0000 UTC m=+68.254460898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls") pod "dns-default-vgmp2" (UID: "74e2ee97-b7ae-45a8-a487-3061cc9bba14") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:44.566228 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:44.566184    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:44.566414 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:44.566281    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:44.566414 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:44.566347    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:44.566542 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:44.566418    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls podName:617ed549-8c3d-45f4-8108-19aaf5d0fa2d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:52.566399851 +0000 UTC m=+60.759427191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbvl8" (UID: "617ed549-8c3d-45f4-8108-19aaf5d0fa2d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:44.566542 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:44.566352    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:44.566542 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:44.566481    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm podName:617ed549-8c3d-45f4-8108-19aaf5d0fa2d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:52.566468234 +0000 UTC m=+60.759495585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hbvl8" (UID: "617ed549-8c3d-45f4-8108-19aaf5d0fa2d") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:48.598350 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:48.598286    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:48.600730 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:48.600707    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls\") pod \"image-registry-85b7597dd7-9qbmm\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") " pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:48.693471 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:48.693433    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:48.821624 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:48.821597    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85b7597dd7-9qbmm"]
Jul 29 10:29:48.825435 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:29:48.825407    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631e8b8d_9ada_4a4b_b163_4d7c83311599.slice/crio-a385a2e29f0bd08fbd582bc8426f02b04c29b91e88b01ae8ba84c6326bcaeb8c WatchSource:0}: Error finding container a385a2e29f0bd08fbd582bc8426f02b04c29b91e88b01ae8ba84c6326bcaeb8c: Status 404 returned error can't find the container with id a385a2e29f0bd08fbd582bc8426f02b04c29b91e88b01ae8ba84c6326bcaeb8c
Jul 29 10:29:49.505891 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:49.505854    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm" event={"ID":"631e8b8d-9ada-4a4b-b163-4d7c83311599","Type":"ContainerStarted","Data":"222fc983eed48950e6d29ce9b5f4feda914fb5079f42144ce6793ae5cc03aa44"}
Jul 29 10:29:49.506044 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:49.505910    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm" event={"ID":"631e8b8d-9ada-4a4b-b163-4d7c83311599","Type":"ContainerStarted","Data":"a385a2e29f0bd08fbd582bc8426f02b04c29b91e88b01ae8ba84c6326bcaeb8c"}
Jul 29 10:29:49.506044 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:49.505995    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:29:49.530444 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:49.528872    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm" podStartSLOduration=17.528850507 podStartE2EDuration="17.528850507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:49.526168834 +0000 UTC m=+57.719196183" watchObservedRunningTime="2026-07-29 10:29:49.528850507 +0000 UTC m=+57.721877853"
Jul 29 10:29:52.632439 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:52.632400    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:52.632850 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:52.632576    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:52.632850 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:52.632585    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:52.632850 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:29:52.632654    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls podName:617ed549-8c3d-45f4-8108-19aaf5d0fa2d nodeName:}" failed. No retries permitted until 2026-07-29 10:30:08.632633826 +0000 UTC m=+76.825661166 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbvl8" (UID: "617ed549-8c3d-45f4-8108-19aaf5d0fa2d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:52.633120 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:52.633105    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:29:53.432851 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:53.432822    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bwrjz"
Jul 29 10:29:57.975724 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:57.975687    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:57.979048 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:57.979030    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 10:29:57.988482 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:57.988463    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdea5088-94eb-4938-938f-ba83acfaa9d2-original-pull-secret\") pod \"global-pull-secret-syncer-nh6vt\" (UID: \"cdea5088-94eb-4938-938f-ba83acfaa9d2\") " pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:58.062766 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.062735    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nh6vt"
Jul 29 10:29:58.076180 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.076149    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs\") pod \"network-metrics-daemon-4wb9p\" (UID: \"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29\") " pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:58.076315 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.076192    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zkxc\" (UniqueName: \"kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc\") pod \"network-check-target-fmxtb\" (UID: \"6a230e54-bb1a-4893-abb4-798a229d4d9d\") " pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:58.079774 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.079753    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 10:29:58.079893 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.079753    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 10:29:58.089233 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.089211    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29-metrics-certs\") pod \"network-metrics-daemon-4wb9p\" (UID: \"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29\") " pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:58.090090 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.090072    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 10:29:58.099782 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.099764    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zkxc\" (UniqueName: \"kubernetes.io/projected/6a230e54-bb1a-4893-abb4-798a229d4d9d-kube-api-access-8zkxc\") pod \"network-check-target-fmxtb\" (UID: \"6a230e54-bb1a-4893-abb4-798a229d4d9d\") " pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:58.193977 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.193950    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nh6vt"]
Jul 29 10:29:58.196819 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:29:58.196795    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdea5088_94eb_4938_938f_ba83acfaa9d2.slice/crio-899cbe945c89ca99737cc988d655b80f7b36ee2856113cbed89b28cb00ffa489 WatchSource:0}: Error finding container 899cbe945c89ca99737cc988d655b80f7b36ee2856113cbed89b28cb00ffa489: Status 404 returned error can't find the container with id 899cbe945c89ca99737cc988d655b80f7b36ee2856113cbed89b28cb00ffa489
Jul 29 10:29:58.372085 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.372054    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kqgtv\""
Jul 29 10:29:58.377859 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.377840    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h2t8s\""
Jul 29 10:29:58.379065 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.379049    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4wb9p"
Jul 29 10:29:58.385590 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.385575    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:29:58.505237 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.505209    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4wb9p"]
Jul 29 10:29:58.508376 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:29:58.508344    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a1f5e6_dc5b_41b1_af1f_c2fa5d510f29.slice/crio-437cdd952b26e52dae6535036d7d556a709c23d0dd61685a6094ecfd27f0cc0c WatchSource:0}: Error finding container 437cdd952b26e52dae6535036d7d556a709c23d0dd61685a6094ecfd27f0cc0c: Status 404 returned error can't find the container with id 437cdd952b26e52dae6535036d7d556a709c23d0dd61685a6094ecfd27f0cc0c
Jul 29 10:29:58.522549 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.522522    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fmxtb"]
Jul 29 10:29:58.525260 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:29:58.525228    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a230e54_bb1a_4893_abb4_798a229d4d9d.slice/crio-c898385c6641abc18ce5dbdd87f6c23339c62cc7f9a08e5070f235f105726eaf WatchSource:0}: Error finding container c898385c6641abc18ce5dbdd87f6c23339c62cc7f9a08e5070f235f105726eaf: Status 404 returned error can't find the container with id c898385c6641abc18ce5dbdd87f6c23339c62cc7f9a08e5070f235f105726eaf
Jul 29 10:29:58.538743 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.538714    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fmxtb" event={"ID":"6a230e54-bb1a-4893-abb4-798a229d4d9d","Type":"ContainerStarted","Data":"c898385c6641abc18ce5dbdd87f6c23339c62cc7f9a08e5070f235f105726eaf"}
Jul 29 10:29:58.539731 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.539708    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4wb9p" event={"ID":"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29","Type":"ContainerStarted","Data":"437cdd952b26e52dae6535036d7d556a709c23d0dd61685a6094ecfd27f0cc0c"}
Jul 29 10:29:58.540744 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:58.540725    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nh6vt" event={"ID":"cdea5088-94eb-4938-938f-ba83acfaa9d2","Type":"ContainerStarted","Data":"899cbe945c89ca99737cc988d655b80f7b36ee2856113cbed89b28cb00ffa489"}
Jul 29 10:29:59.120546 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.120444    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"]
Jul 29 10:29:59.123696 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.123656    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.127247 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.127213    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 29 10:29:59.128887 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.128827    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 29 10:29:59.128887 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.128863    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 29 10:29:59.129096 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.129075    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 10:29:59.129217 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.129131    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 10:29:59.129277 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.129264    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 10:29:59.129398 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.129382    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 29 10:29:59.140843 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.140794    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"]
Jul 29 10:29:59.170663 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.170626    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85b7597dd7-9qbmm"]
Jul 29 10:29:59.186406 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.186371    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.186578 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.186412    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-hub\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.186578 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.186443    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znwp7\" (UniqueName: \"kubernetes.io/projected/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-kube-api-access-znwp7\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.186578 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.186471    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.186762 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.186631    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.186762 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.186743    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-ca\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.287356 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.287323    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.287511 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.287398    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-ca\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.287593 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.287526    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.287593 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.287556    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-hub\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.287735 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.287600    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znwp7\" (UniqueName: \"kubernetes.io/projected/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-kube-api-access-znwp7\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.287735 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.287634    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.288181 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.288155    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.290815 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.290773    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-ca\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.290815 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.290781    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-hub\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.291218 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.291198    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.292363 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.292340    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.300768 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.300747    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znwp7\" (UniqueName: \"kubernetes.io/projected/c009d3e2-e571-44be-bfd4-7cd0f399d3a1-kube-api-access-znwp7\") pod \"cluster-proxy-proxy-agent-55bf4b59d-k4k6m\" (UID: \"c009d3e2-e571-44be-bfd4-7cd0f399d3a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.447770 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.447671    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"
Jul 29 10:29:59.973597 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:29:59.973536    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m"]
Jul 29 10:30:00.094326 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:00.094247    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:30:00.097810 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:00.097776    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74e2ee97-b7ae-45a8-a487-3061cc9bba14-metrics-tls\") pod \"dns-default-vgmp2\" (UID: \"74e2ee97-b7ae-45a8-a487-3061cc9bba14\") " pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:30:00.113208 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:30:00.113174    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc009d3e2_e571_44be_bfd4_7cd0f399d3a1.slice/crio-17915ad91a1ab6bc0bcd2b375f5a41b093483dc67644de24e29a912f9f59df3f WatchSource:0}: Error finding container 17915ad91a1ab6bc0bcd2b375f5a41b093483dc67644de24e29a912f9f59df3f: Status 404 returned error can't find the container with id 17915ad91a1ab6bc0bcd2b375f5a41b093483dc67644de24e29a912f9f59df3f
Jul 29 10:30:00.360466 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:00.360237    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9rdn4\""
Jul 29 10:30:00.367880 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:00.367858    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:30:00.547407 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:00.547360    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vgmp2"]
Jul 29 10:30:00.549698 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:30:00.549648    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e2ee97_b7ae_45a8_a487_3061cc9bba14.slice/crio-2fb6209afd258dca61c83433738604df5da81c7cc0395c851c328273e5c6a74f WatchSource:0}: Error finding container 2fb6209afd258dca61c83433738604df5da81c7cc0395c851c328273e5c6a74f: Status 404 returned error can't find the container with id 2fb6209afd258dca61c83433738604df5da81c7cc0395c851c328273e5c6a74f
Jul 29 10:30:00.551619 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:00.551578    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m" event={"ID":"c009d3e2-e571-44be-bfd4-7cd0f399d3a1","Type":"ContainerStarted","Data":"17915ad91a1ab6bc0bcd2b375f5a41b093483dc67644de24e29a912f9f59df3f"}
Jul 29 10:30:00.553994 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:00.553910    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4wb9p" event={"ID":"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29","Type":"ContainerStarted","Data":"0c5374bd62abd5cbad634bc58e4cbab8bd7e2c7b411976f7b11f0cbaef2c122d"}
Jul 29 10:30:00.553994 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:00.553963    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4wb9p" event={"ID":"06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29","Type":"ContainerStarted","Data":"0bbfc51b0ef5be93988b7e9b9c680f1586bcba1343da111d04276ad382c54a99"}
Jul 29 10:30:00.575094 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:00.575037    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4wb9p" podStartSLOduration=65.948717026 podStartE2EDuration="1m7.57501644s" totalImagesPullingTime="1.626299414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:00.573972649 +0000 UTC m=+68.766999996" watchObservedRunningTime="2026-07-29 10:30:00.57501644 +0000 UTC m=+68.768043789"
Jul 29 10:30:01.558575 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:01.558537    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vgmp2" event={"ID":"74e2ee97-b7ae-45a8-a487-3061cc9bba14","Type":"ContainerStarted","Data":"2fb6209afd258dca61c83433738604df5da81c7cc0395c851c328273e5c6a74f"}
Jul 29 10:30:03.571454 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:03.570651    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nh6vt" event={"ID":"cdea5088-94eb-4938-938f-ba83acfaa9d2","Type":"ContainerStarted","Data":"c2d71b1efd7deb7a4f3193c4fc094ec21c10ed53476ac9380964f5faa937c78f"}
Jul 29 10:30:03.574130 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:03.574080    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fmxtb" event={"ID":"6a230e54-bb1a-4893-abb4-798a229d4d9d","Type":"ContainerStarted","Data":"9056e270a1b5b073d02a7ebb50f51ce50da2cbef05529fd7fdaf844f5f570705"}
Jul 29 10:30:03.574304 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:03.574287    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:30:03.589406 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:03.589293    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nh6vt" podStartSLOduration=65.454736017 podStartE2EDuration="1m10.589273798s" totalImagesPullingTime="5.134537781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:03.587971983 +0000 UTC m=+71.780999343" watchObservedRunningTime="2026-07-29 10:30:03.589273798 +0000 UTC m=+71.782301147"
Jul 29 10:30:03.605489 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:03.605350    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fmxtb" podStartSLOduration=65.803826857 podStartE2EDuration="1m10.605328755s" totalImagesPullingTime="4.801501898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:03.604826957 +0000 UTC m=+71.797854305" watchObservedRunningTime="2026-07-29 10:30:03.605328755 +0000 UTC m=+71.798356104"
Jul 29 10:30:07.592733 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:07.592693    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m" event={"ID":"c009d3e2-e571-44be-bfd4-7cd0f399d3a1","Type":"ContainerStarted","Data":"e1e9cdc27ccec6fad30d182777920a6ddc5b90c058b0c0ee8f3ab4d4770f69f8"}
Jul 29 10:30:07.592733 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:07.592738    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m" event={"ID":"c009d3e2-e571-44be-bfd4-7cd0f399d3a1","Type":"ContainerStarted","Data":"ceec660f9b35da16b96c2a84017e99f430b734ef6d065b4f53fc40f32a6c4e33"}
Jul 29 10:30:07.593298 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:07.592750    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m" event={"ID":"c009d3e2-e571-44be-bfd4-7cd0f399d3a1","Type":"ContainerStarted","Data":"26b3f5f3b5b15c7e30fafe81d5ad77795c8e54c45b9c8961d71d30fef930c9a0"}
Jul 29 10:30:07.594420 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:07.594400    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vgmp2" event={"ID":"74e2ee97-b7ae-45a8-a487-3061cc9bba14","Type":"ContainerStarted","Data":"45b2aa06dfd7050bf7fbe88728e8d4213c9c3447163ce94f505c3a67aa1fbab6"}
Jul 29 10:30:07.594510 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:07.594438    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vgmp2" event={"ID":"74e2ee97-b7ae-45a8-a487-3061cc9bba14","Type":"ContainerStarted","Data":"f4bcda8bb29dc4cbccd6c81a49a7e6cd6ea7161dc719a25565cef951391c8ae1"}
Jul 29 10:30:07.594568 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:07.594528    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:30:07.618536 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:07.618479    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55bf4b59d-k4k6m" podStartSLOduration=2.220674058 podStartE2EDuration="8.618463155s" totalImagesPullingTime="6.397789097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:07.618043247 +0000 UTC m=+75.811070594" watchObservedRunningTime="2026-07-29 10:30:07.618463155 +0000 UTC m=+75.811490501"
Jul 29 10:30:07.637999 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:07.637949    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vgmp2" podStartSLOduration=33.72404862 podStartE2EDuration="39.6379318s" totalImagesPullingTime="5.91388318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:07.636988978 +0000 UTC m=+75.830016325" watchObservedRunningTime="2026-07-29 10:30:07.6379318 +0000 UTC m=+75.830959212"
Jul 29 10:30:08.670584 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:08.670543    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:30:08.672740 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:08.672714    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/617ed549-8c3d-45f4-8108-19aaf5d0fa2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbvl8\" (UID: \"617ed549-8c3d-45f4-8108-19aaf5d0fa2d\") " pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:30:08.686451 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:08.686428    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l6k5g\""
Jul 29 10:30:08.694045 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:08.694019    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hbvl8"
Jul 29 10:30:08.825301 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:08.825275    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hbvl8"]
Jul 29 10:30:08.827650 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:30:08.827624    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617ed549_8c3d_45f4_8108_19aaf5d0fa2d.slice/crio-a57d245f9da4d1192c45a74a1a2d6b2271c88956652e22814a5b6635f6c80563 WatchSource:0}: Error finding container a57d245f9da4d1192c45a74a1a2d6b2271c88956652e22814a5b6635f6c80563: Status 404 returned error can't find the container with id a57d245f9da4d1192c45a74a1a2d6b2271c88956652e22814a5b6635f6c80563
Jul 29 10:30:09.176449 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:09.176415    2584 patch_prober.go:28] interesting pod/image-registry-85b7597dd7-9qbmm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 29 10:30:09.176606 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:09.176467    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm" podUID="631e8b8d-9ada-4a4b-b163-4d7c83311599" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:30:09.603692 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:09.603644    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hbvl8" event={"ID":"617ed549-8c3d-45f4-8108-19aaf5d0fa2d","Type":"ContainerStarted","Data":"a57d245f9da4d1192c45a74a1a2d6b2271c88956652e22814a5b6635f6c80563"}
Jul 29 10:30:14.625837 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:14.625796    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hbvl8" event={"ID":"617ed549-8c3d-45f4-8108-19aaf5d0fa2d","Type":"ContainerStarted","Data":"80b2a58d17fd3bb20e6362d5ad6286d33448039d08dc6077bc2946974cc0ed97"}
Jul 29 10:30:17.598956 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:17.598925    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vgmp2"
Jul 29 10:30:17.638167 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:17.638134    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hbvl8" event={"ID":"617ed549-8c3d-45f4-8108-19aaf5d0fa2d","Type":"ContainerStarted","Data":"80919036cc44dcb25bf9249a82607ba5796ad750c7993580d27873d3fec9949e"}
Jul 29 10:30:19.175978 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:19.175952    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:30:21.653600 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:21.653566    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hbvl8" event={"ID":"617ed549-8c3d-45f4-8108-19aaf5d0fa2d","Type":"ContainerStarted","Data":"78d5836d1fe035f9370bcae7428f87d74c84fb8c8ce6cba4dd563b69da17bbfa"}
Jul 29 10:30:21.674689 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:21.674637    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hbvl8" podStartSLOduration=33.827692454 podStartE2EDuration="45.674617033s" totalImagesPullingTime="11.846924579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:21.673342262 +0000 UTC m=+89.866369609" watchObservedRunningTime="2026-07-29 10:30:21.674617033 +0000 UTC m=+89.867644379"
Jul 29 10:30:24.197071 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.197013    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm" podUID="631e8b8d-9ada-4a4b-b163-4d7c83311599" containerName="registry" containerID="cri-o://222fc983eed48950e6d29ce9b5f4feda914fb5079f42144ce6793ae5cc03aa44" gracePeriod=30
Jul 29 10:30:24.443005 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.442978    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:30:24.592926 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.592895    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/631e8b8d-9ada-4a4b-b163-4d7c83311599-installation-pull-secrets\") pod \"631e8b8d-9ada-4a4b-b163-4d7c83311599\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") "
Jul 29 10:30:24.593068 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.592941    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2l7l\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-kube-api-access-t2l7l\") pod \"631e8b8d-9ada-4a4b-b163-4d7c83311599\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") "
Jul 29 10:30:24.593068 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.592960    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-certificates\") pod \"631e8b8d-9ada-4a4b-b163-4d7c83311599\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") "
Jul 29 10:30:24.593068 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.592985    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/631e8b8d-9ada-4a4b-b163-4d7c83311599-ca-trust-extracted\") pod \"631e8b8d-9ada-4a4b-b163-4d7c83311599\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") "
Jul 29 10:30:24.593068 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.593013    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/631e8b8d-9ada-4a4b-b163-4d7c83311599-image-registry-private-configuration\") pod \"631e8b8d-9ada-4a4b-b163-4d7c83311599\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") "
Jul 29 10:30:24.593068 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.593033    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls\") pod \"631e8b8d-9ada-4a4b-b163-4d7c83311599\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") "
Jul 29 10:30:24.593068 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.593064    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/631e8b8d-9ada-4a4b-b163-4d7c83311599-trusted-ca\") pod \"631e8b8d-9ada-4a4b-b163-4d7c83311599\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") "
Jul 29 10:30:24.593339 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.593096    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-bound-sa-token\") pod \"631e8b8d-9ada-4a4b-b163-4d7c83311599\" (UID: \"631e8b8d-9ada-4a4b-b163-4d7c83311599\") "
Jul 29 10:30:24.593722 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.593636    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "631e8b8d-9ada-4a4b-b163-4d7c83311599" (UID: "631e8b8d-9ada-4a4b-b163-4d7c83311599"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:30:24.593856 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.593826    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631e8b8d-9ada-4a4b-b163-4d7c83311599-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "631e8b8d-9ada-4a4b-b163-4d7c83311599" (UID: "631e8b8d-9ada-4a4b-b163-4d7c83311599"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:30:24.595696 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.595650    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-kube-api-access-t2l7l" (OuterVolumeSpecName: "kube-api-access-t2l7l") pod "631e8b8d-9ada-4a4b-b163-4d7c83311599" (UID: "631e8b8d-9ada-4a4b-b163-4d7c83311599"). InnerVolumeSpecName "kube-api-access-t2l7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:30:24.595818 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.595719    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "631e8b8d-9ada-4a4b-b163-4d7c83311599" (UID: "631e8b8d-9ada-4a4b-b163-4d7c83311599"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:30:24.595935 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.595859    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631e8b8d-9ada-4a4b-b163-4d7c83311599-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "631e8b8d-9ada-4a4b-b163-4d7c83311599" (UID: "631e8b8d-9ada-4a4b-b163-4d7c83311599"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:30:24.595935 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.595875    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "631e8b8d-9ada-4a4b-b163-4d7c83311599" (UID: "631e8b8d-9ada-4a4b-b163-4d7c83311599"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:30:24.596041 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.595938    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631e8b8d-9ada-4a4b-b163-4d7c83311599-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "631e8b8d-9ada-4a4b-b163-4d7c83311599" (UID: "631e8b8d-9ada-4a4b-b163-4d7c83311599"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:30:24.602204 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.602182    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631e8b8d-9ada-4a4b-b163-4d7c83311599-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "631e8b8d-9ada-4a4b-b163-4d7c83311599" (UID: "631e8b8d-9ada-4a4b-b163-4d7c83311599"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:30:24.664026 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.663994    2584 generic.go:356] "Generic (PLEG): container finished" podID="631e8b8d-9ada-4a4b-b163-4d7c83311599" containerID="222fc983eed48950e6d29ce9b5f4feda914fb5079f42144ce6793ae5cc03aa44" exitCode=0
Jul 29 10:30:24.664177 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.664040    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm"
Jul 29 10:30:24.664177 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.664041    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm" event={"ID":"631e8b8d-9ada-4a4b-b163-4d7c83311599","Type":"ContainerDied","Data":"222fc983eed48950e6d29ce9b5f4feda914fb5079f42144ce6793ae5cc03aa44"}
Jul 29 10:30:24.664247 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.664178    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b7597dd7-9qbmm" event={"ID":"631e8b8d-9ada-4a4b-b163-4d7c83311599","Type":"ContainerDied","Data":"a385a2e29f0bd08fbd582bc8426f02b04c29b91e88b01ae8ba84c6326bcaeb8c"}
Jul 29 10:30:24.664247 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.664210    2584 scope.go:117] "RemoveContainer" containerID="222fc983eed48950e6d29ce9b5f4feda914fb5079f42144ce6793ae5cc03aa44"
Jul 29 10:30:24.676778 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.676760    2584 scope.go:117] "RemoveContainer" containerID="222fc983eed48950e6d29ce9b5f4feda914fb5079f42144ce6793ae5cc03aa44"
Jul 29 10:30:24.677060 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:30:24.677038    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"222fc983eed48950e6d29ce9b5f4feda914fb5079f42144ce6793ae5cc03aa44\": container with ID starting with 222fc983eed48950e6d29ce9b5f4feda914fb5079f42144ce6793ae5cc03aa44 not found: ID does not exist" containerID="222fc983eed48950e6d29ce9b5f4feda914fb5079f42144ce6793ae5cc03aa44"
Jul 29 10:30:24.677113 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.677068    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"222fc983eed48950e6d29ce9b5f4feda914fb5079f42144ce6793ae5cc03aa44"} err="failed to get container status \"222fc983eed48950e6d29ce9b5f4feda914fb5079f42144ce6793ae5cc03aa44\": rpc error: code = NotFound desc = could not find container \"222fc983eed48950e6d29ce9b5f4feda914fb5079f42144ce6793ae5cc03aa44\": container with ID starting with 222fc983eed48950e6d29ce9b5f4feda914fb5079f42144ce6793ae5cc03aa44 not found: ID does not exist"
Jul 29 10:30:24.687398 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.687374    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85b7597dd7-9qbmm"]
Jul 29 10:30:24.694441 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.694419    2584 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/631e8b8d-9ada-4a4b-b163-4d7c83311599-ca-trust-extracted\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:30:24.694545 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.694446    2584 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/631e8b8d-9ada-4a4b-b163-4d7c83311599-image-registry-private-configuration\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:30:24.694545 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.694461    2584 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-tls\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:30:24.694545 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.694473    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/631e8b8d-9ada-4a4b-b163-4d7c83311599-trusted-ca\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:30:24.694545 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.694484    2584 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-bound-sa-token\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:30:24.694545 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.694495    2584 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/631e8b8d-9ada-4a4b-b163-4d7c83311599-installation-pull-secrets\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:30:24.694545 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.694510    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2l7l\" (UniqueName: \"kubernetes.io/projected/631e8b8d-9ada-4a4b-b163-4d7c83311599-kube-api-access-t2l7l\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:30:24.694545 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.694522    2584 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/631e8b8d-9ada-4a4b-b163-4d7c83311599-registry-certificates\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:30:24.696402 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:24.696382    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85b7597dd7-9qbmm"]
Jul 29 10:30:25.086158 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.086128    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"]
Jul 29 10:30:25.086387 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.086375    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="631e8b8d-9ada-4a4b-b163-4d7c83311599" containerName="registry"
Jul 29 10:30:25.086435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.086388    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="631e8b8d-9ada-4a4b-b163-4d7c83311599" containerName="registry"
Jul 29 10:30:25.086435 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.086432    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="631e8b8d-9ada-4a4b-b163-4d7c83311599" containerName="registry"
Jul 29 10:30:25.091288 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.091269    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"
Jul 29 10:30:25.094716 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.094694    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 29 10:30:25.094812 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.094757    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 10:30:25.094812 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.094768    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 10:30:25.095024 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.095010    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 29 10:30:25.095945 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.095931    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gp9tv\""
Jul 29 10:30:25.096029 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.095995    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 10:30:25.101595 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.101448    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"]
Jul 29 10:30:25.128048 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.128019    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d7tx4"]
Jul 29 10:30:25.131088 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.131069    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.137511 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.137492    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nhtns\""
Jul 29 10:30:25.137605 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.137531    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 10:30:25.137730 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.137717    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 10:30:25.137793 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.137742    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 10:30:25.197182 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.197150    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d27cf8b-53cf-45c5-8aba-302fd81d2364-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-rsxwn\" (UID: \"3d27cf8b-53cf-45c5-8aba-302fd81d2364\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"
Jul 29 10:30:25.197589 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.197193    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d27cf8b-53cf-45c5-8aba-302fd81d2364-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-rsxwn\" (UID: \"3d27cf8b-53cf-45c5-8aba-302fd81d2364\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"
Jul 29 10:30:25.197589 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.197221    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d27cf8b-53cf-45c5-8aba-302fd81d2364-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rsxwn\" (UID: \"3d27cf8b-53cf-45c5-8aba-302fd81d2364\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"
Jul 29 10:30:25.197589 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.197276    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnhvb\" (UniqueName: \"kubernetes.io/projected/3d27cf8b-53cf-45c5-8aba-302fd81d2364-kube-api-access-fnhvb\") pod \"openshift-state-metrics-b8f49f5c6-rsxwn\" (UID: \"3d27cf8b-53cf-45c5-8aba-302fd81d2364\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"
Jul 29 10:30:25.298030 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.298000    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d27cf8b-53cf-45c5-8aba-302fd81d2364-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rsxwn\" (UID: \"3d27cf8b-53cf-45c5-8aba-302fd81d2364\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"
Jul 29 10:30:25.298030 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.298032    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebff90ce-6282-4f61-a34d-9164b7c382ea-root\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.298247 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.298052    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebff90ce-6282-4f61-a34d-9164b7c382ea-node-exporter-wtmp\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.298247 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.298073    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnhvb\" (UniqueName: \"kubernetes.io/projected/3d27cf8b-53cf-45c5-8aba-302fd81d2364-kube-api-access-fnhvb\") pod \"openshift-state-metrics-b8f49f5c6-rsxwn\" (UID: \"3d27cf8b-53cf-45c5-8aba-302fd81d2364\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"
Jul 29 10:30:25.298247 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.298090    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebff90ce-6282-4f61-a34d-9164b7c382ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.298247 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.298131    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebff90ce-6282-4f61-a34d-9164b7c382ea-node-exporter-tls\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.298247 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:30:25.298158    2584 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 29 10:30:25.298247 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:30:25.298216    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d27cf8b-53cf-45c5-8aba-302fd81d2364-openshift-state-metrics-tls podName:3d27cf8b-53cf-45c5-8aba-302fd81d2364 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:25.798199785 +0000 UTC m=+93.991227109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3d27cf8b-53cf-45c5-8aba-302fd81d2364-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-rsxwn" (UID: "3d27cf8b-53cf-45c5-8aba-302fd81d2364") : secret "openshift-state-metrics-tls" not found
Jul 29 10:30:25.298438 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.298300    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebff90ce-6282-4f61-a34d-9164b7c382ea-node-exporter-textfile\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.298438 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.298336    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d27cf8b-53cf-45c5-8aba-302fd81d2364-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-rsxwn\" (UID: \"3d27cf8b-53cf-45c5-8aba-302fd81d2364\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"
Jul 29 10:30:25.298438 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.298360    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gb9x\" (UniqueName: \"kubernetes.io/projected/ebff90ce-6282-4f61-a34d-9164b7c382ea-kube-api-access-8gb9x\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.298438 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.298400    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebff90ce-6282-4f61-a34d-9164b7c382ea-metrics-client-ca\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.298438 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.298429    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d27cf8b-53cf-45c5-8aba-302fd81d2364-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-rsxwn\" (UID: \"3d27cf8b-53cf-45c5-8aba-302fd81d2364\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"
Jul 29 10:30:25.298614 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.298455    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebff90ce-6282-4f61-a34d-9164b7c382ea-sys\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.298614 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.298488    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebff90ce-6282-4f61-a34d-9164b7c382ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.299076 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.299056    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d27cf8b-53cf-45c5-8aba-302fd81d2364-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-rsxwn\" (UID: \"3d27cf8b-53cf-45c5-8aba-302fd81d2364\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"
Jul 29 10:30:25.300938 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.300918    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d27cf8b-53cf-45c5-8aba-302fd81d2364-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-rsxwn\" (UID: \"3d27cf8b-53cf-45c5-8aba-302fd81d2364\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"
Jul 29 10:30:25.314316 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.314284    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnhvb\" (UniqueName: \"kubernetes.io/projected/3d27cf8b-53cf-45c5-8aba-302fd81d2364-kube-api-access-fnhvb\") pod \"openshift-state-metrics-b8f49f5c6-rsxwn\" (UID: \"3d27cf8b-53cf-45c5-8aba-302fd81d2364\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"
Jul 29 10:30:25.399170 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.399086    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebff90ce-6282-4f61-a34d-9164b7c382ea-metrics-client-ca\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.399170 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.399120    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebff90ce-6282-4f61-a34d-9164b7c382ea-sys\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.399170 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.399142    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebff90ce-6282-4f61-a34d-9164b7c382ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.399170 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.399174    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebff90ce-6282-4f61-a34d-9164b7c382ea-root\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.399493 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.399196    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebff90ce-6282-4f61-a34d-9164b7c382ea-node-exporter-wtmp\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.399493 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.399223    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebff90ce-6282-4f61-a34d-9164b7c382ea-sys\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.399493 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.399232    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebff90ce-6282-4f61-a34d-9164b7c382ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.399493 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.399245    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebff90ce-6282-4f61-a34d-9164b7c382ea-root\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.399493 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.399262    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebff90ce-6282-4f61-a34d-9164b7c382ea-node-exporter-tls\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.399493 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.399291    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebff90ce-6282-4f61-a34d-9164b7c382ea-node-exporter-textfile\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.399493 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.399334    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gb9x\" (UniqueName: \"kubernetes.io/projected/ebff90ce-6282-4f61-a34d-9164b7c382ea-kube-api-access-8gb9x\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.399493 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.399378    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebff90ce-6282-4f61-a34d-9164b7c382ea-node-exporter-wtmp\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.399897 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.399660    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebff90ce-6282-4f61-a34d-9164b7c382ea-node-exporter-textfile\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.399897 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.399763    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebff90ce-6282-4f61-a34d-9164b7c382ea-metrics-client-ca\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.399897 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.399763    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebff90ce-6282-4f61-a34d-9164b7c382ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.401561 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.401540    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebff90ce-6282-4f61-a34d-9164b7c382ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.401755 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.401738    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebff90ce-6282-4f61-a34d-9164b7c382ea-node-exporter-tls\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.408288 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.408268    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gb9x\" (UniqueName: \"kubernetes.io/projected/ebff90ce-6282-4f61-a34d-9164b7c382ea-kube-api-access-8gb9x\") pod \"node-exporter-d7tx4\" (UID: \"ebff90ce-6282-4f61-a34d-9164b7c382ea\") " pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.439254 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.439227    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d7tx4"
Jul 29 10:30:25.450385 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:30:25.450357    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebff90ce_6282_4f61_a34d_9164b7c382ea.slice/crio-38bc0749da0fa780680db8b48f0fa2acaf48754209372dd9e5efadb942c4c80e WatchSource:0}: Error finding container 38bc0749da0fa780680db8b48f0fa2acaf48754209372dd9e5efadb942c4c80e: Status 404 returned error can't find the container with id 38bc0749da0fa780680db8b48f0fa2acaf48754209372dd9e5efadb942c4c80e
Jul 29 10:30:25.669043 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.668950    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7tx4" event={"ID":"ebff90ce-6282-4f61-a34d-9164b7c382ea","Type":"ContainerStarted","Data":"38bc0749da0fa780680db8b48f0fa2acaf48754209372dd9e5efadb942c4c80e"}
Jul 29 10:30:25.801801 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.801767    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d27cf8b-53cf-45c5-8aba-302fd81d2364-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rsxwn\" (UID: \"3d27cf8b-53cf-45c5-8aba-302fd81d2364\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"
Jul 29 10:30:25.804064 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.804046    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d27cf8b-53cf-45c5-8aba-302fd81d2364-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rsxwn\" (UID: \"3d27cf8b-53cf-45c5-8aba-302fd81d2364\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"
Jul 29 10:30:25.999455 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:25.999368    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"
Jul 29 10:30:26.156033 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.155998    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn"]
Jul 29 10:30:26.160230 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:30:26.160203    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d27cf8b_53cf_45c5_8aba_302fd81d2364.slice/crio-447f3e4a5e517460ad4ae8dd0506ef1de2be3f48e4dc7da14d599f86a05839d3 WatchSource:0}: Error finding container 447f3e4a5e517460ad4ae8dd0506ef1de2be3f48e4dc7da14d599f86a05839d3: Status 404 returned error can't find the container with id 447f3e4a5e517460ad4ae8dd0506ef1de2be3f48e4dc7da14d599f86a05839d3
Jul 29 10:30:26.160902 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.160881    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 10:30:26.166126 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.166104    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.169129 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.169088    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 29 10:30:26.169129 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.169119    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 29 10:30:26.169283 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.169265    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-d4wdm\""
Jul 29 10:30:26.169343 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.169094    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 29 10:30:26.169409 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.169390    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 29 10:30:26.169555 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.169535    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 29 10:30:26.169634 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.169591    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 29 10:30:26.169707 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.169628    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 29 10:30:26.169936 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.169916    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 29 10:30:26.170582 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.170409    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 29 10:30:26.186291 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.186270    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 10:30:26.205568 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.205544    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/040367c1-1b23-442f-b75d-9df47bb63a34-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.205901 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.205575    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/040367c1-1b23-442f-b75d-9df47bb63a34-config-out\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.205901 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.205602    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.205901 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.205666    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/040367c1-1b23-442f-b75d-9df47bb63a34-tls-assets\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.205901 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.205708    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.205901 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.205730    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-web-config\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.205901 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.205753    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-config-volume\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.205901 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.205767    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.205901 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.205804    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.205901 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.205824    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jhlv\" (UniqueName: \"kubernetes.io/projected/040367c1-1b23-442f-b75d-9df47bb63a34-kube-api-access-2jhlv\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.205901 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.205852    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/040367c1-1b23-442f-b75d-9df47bb63a34-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.205901 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.205878    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.205901 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.205897    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/040367c1-1b23-442f-b75d-9df47bb63a34-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.255490 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.255409    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="631e8b8d-9ada-4a4b-b163-4d7c83311599" path="/var/lib/kubelet/pods/631e8b8d-9ada-4a4b-b163-4d7c83311599/volumes"
Jul 29 10:30:26.307593 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.307556    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/040367c1-1b23-442f-b75d-9df47bb63a34-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.307792 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.307611    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/040367c1-1b23-442f-b75d-9df47bb63a34-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.307792 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.307651    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/040367c1-1b23-442f-b75d-9df47bb63a34-config-out\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.307792 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.307729    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.307792 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.307770    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/040367c1-1b23-442f-b75d-9df47bb63a34-tls-assets\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.307985 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.307799    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.307985 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.307830    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-web-config\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.307985 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.307874    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-config-volume\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.307985 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.307909    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.307985 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.307955    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.308200 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.307994    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jhlv\" (UniqueName: \"kubernetes.io/projected/040367c1-1b23-442f-b75d-9df47bb63a34-kube-api-access-2jhlv\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.308200 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.308042    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/040367c1-1b23-442f-b75d-9df47bb63a34-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.308200 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.308078    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.308337 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:30:26.308260    2584 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 29 10:30:26.308381 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:30:26.308347    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-secret-alertmanager-main-tls podName:040367c1-1b23-442f-b75d-9df47bb63a34 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:26.808316658 +0000 UTC m=+95.001343998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "040367c1-1b23-442f-b75d-9df47bb63a34") : secret "alertmanager-main-tls" not found
Jul 29 10:30:26.309533 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:30:26.309250    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/040367c1-1b23-442f-b75d-9df47bb63a34-alertmanager-trusted-ca-bundle podName:040367c1-1b23-442f-b75d-9df47bb63a34 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:26.809231383 +0000 UTC m=+95.002258715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/040367c1-1b23-442f-b75d-9df47bb63a34-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "040367c1-1b23-442f-b75d-9df47bb63a34") : configmap references non-existent config key: ca-bundle.crt
Jul 29 10:30:26.310848 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.310404    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/040367c1-1b23-442f-b75d-9df47bb63a34-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.310848 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.310432    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/040367c1-1b23-442f-b75d-9df47bb63a34-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.311689 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.311650    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/040367c1-1b23-442f-b75d-9df47bb63a34-config-out\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.313814 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.313411    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.314736 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.314665    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-web-config\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.315466 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.315440    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.316235 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.316118    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-config-volume\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.316235 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.316204    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.316963 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.316941    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/040367c1-1b23-442f-b75d-9df47bb63a34-tls-assets\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.317647 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.317629    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.319486 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.319465    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jhlv\" (UniqueName: \"kubernetes.io/projected/040367c1-1b23-442f-b75d-9df47bb63a34-kube-api-access-2jhlv\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.673857 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.673822    2584 generic.go:356] "Generic (PLEG): container finished" podID="ebff90ce-6282-4f61-a34d-9164b7c382ea" containerID="37cedcaafce2807eb2790a71dd9d737f16086a47271d34a7b9f0cb568cd9c0ba" exitCode=0
Jul 29 10:30:26.674003 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.673911    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7tx4" event={"ID":"ebff90ce-6282-4f61-a34d-9164b7c382ea","Type":"ContainerDied","Data":"37cedcaafce2807eb2790a71dd9d737f16086a47271d34a7b9f0cb568cd9c0ba"}
Jul 29 10:30:26.675593 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.675570    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn" event={"ID":"3d27cf8b-53cf-45c5-8aba-302fd81d2364","Type":"ContainerStarted","Data":"b7509f961614f0f9036f6793025c3dce7ff04c102229df10207d77bfdf1067dd"}
Jul 29 10:30:26.675712 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.675615    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn" event={"ID":"3d27cf8b-53cf-45c5-8aba-302fd81d2364","Type":"ContainerStarted","Data":"4cb10be967a0c829a9e7fe0d134110e41ef827eb3b24c9210b6d057d7fc5ef94"}
Jul 29 10:30:26.675712 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.675634    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn" event={"ID":"3d27cf8b-53cf-45c5-8aba-302fd81d2364","Type":"ContainerStarted","Data":"447f3e4a5e517460ad4ae8dd0506ef1de2be3f48e4dc7da14d599f86a05839d3"}
Jul 29 10:30:26.811656 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.811624    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/040367c1-1b23-442f-b75d-9df47bb63a34-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.811656 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.811659    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.812401 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.812377    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/040367c1-1b23-442f-b75d-9df47bb63a34-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:26.814205 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:26.814184    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/040367c1-1b23-442f-b75d-9df47bb63a34-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"040367c1-1b23-442f-b75d-9df47bb63a34\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:27.077342 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:27.077306    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:27.225831 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:27.225798    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 10:30:27.326775 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:30:27.326735    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040367c1_1b23_442f_b75d_9df47bb63a34.slice/crio-fd39f180f9e78f810ea215611151060e91d04a50bdd9f6b0a8dad0b5ef012bd0 WatchSource:0}: Error finding container fd39f180f9e78f810ea215611151060e91d04a50bdd9f6b0a8dad0b5ef012bd0: Status 404 returned error can't find the container with id fd39f180f9e78f810ea215611151060e91d04a50bdd9f6b0a8dad0b5ef012bd0
Jul 29 10:30:27.680774 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:27.680662    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"040367c1-1b23-442f-b75d-9df47bb63a34","Type":"ContainerStarted","Data":"fd39f180f9e78f810ea215611151060e91d04a50bdd9f6b0a8dad0b5ef012bd0"}
Jul 29 10:30:27.682777 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:27.682752    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn" event={"ID":"3d27cf8b-53cf-45c5-8aba-302fd81d2364","Type":"ContainerStarted","Data":"d9c30a1dd7977ab03fd1a6fbb4651f7cc9e9641fe1832e9452e163d6fcee785a"}
Jul 29 10:30:27.684790 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:27.684760    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7tx4" event={"ID":"ebff90ce-6282-4f61-a34d-9164b7c382ea","Type":"ContainerStarted","Data":"c8cd5de397c7663fc0c3aafe718815752cab37487f5eab3dca7ab033be152246"}
Jul 29 10:30:27.684884 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:27.684809    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7tx4" event={"ID":"ebff90ce-6282-4f61-a34d-9164b7c382ea","Type":"ContainerStarted","Data":"9a0a57ad5368150e5ef2bf365162b53b3b3ae683f731b749cc58633d596dbf92"}
Jul 29 10:30:27.702260 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:27.702212    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsxwn" podStartSLOduration=1.8217245549999999 podStartE2EDuration="2.702195973s" totalImagesPullingTime="880.471418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:27.700764407 +0000 UTC m=+95.893791754" watchObservedRunningTime="2026-07-29 10:30:27.702195973 +0000 UTC m=+95.895223321"
Jul 29 10:30:27.719984 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:27.719933    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d7tx4" podStartSLOduration=1.632106211 podStartE2EDuration="2.719915055s" totalImagesPullingTime="1.025721761s" totalInitContainerRuntime="62.087083ms" isStatefulPod=true podCreationTimestamp="2026-07-29 10:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:27.719230942 +0000 UTC m=+95.912258289" watchObservedRunningTime="2026-07-29 10:30:27.719915055 +0000 UTC m=+95.912942403"
Jul 29 10:30:28.692184 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:28.692151    2584 generic.go:356] "Generic (PLEG): container finished" podID="040367c1-1b23-442f-b75d-9df47bb63a34" containerID="63d36e46943ce1144326b985aa9465e528aee6ee11b42fa8ef80d1c57e9f8990" exitCode=0
Jul 29 10:30:28.692622 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:28.692232    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"040367c1-1b23-442f-b75d-9df47bb63a34","Type":"ContainerDied","Data":"63d36e46943ce1144326b985aa9465e528aee6ee11b42fa8ef80d1c57e9f8990"}
Jul 29 10:30:29.886827 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:29.886789    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp"]
Jul 29 10:30:29.890127 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:29.890106    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp"
Jul 29 10:30:29.893134 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:29.893113    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 29 10:30:29.895991 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:29.895972    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-99hzv\""
Jul 29 10:30:29.904169 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:29.904147    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp"]
Jul 29 10:30:29.937277 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:29.937244    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4d62d964-dccd-4dcb-9c9f-e07111b74671-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-d88rp\" (UID: \"4d62d964-dccd-4dcb-9c9f-e07111b74671\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp"
Jul 29 10:30:29.937428 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:29.937295    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbhmm\" (UniqueName: \"kubernetes.io/projected/4d62d964-dccd-4dcb-9c9f-e07111b74671-kube-api-access-xbhmm\") pod \"monitoring-plugin-68b7fc9574-d88rp\" (UID: \"4d62d964-dccd-4dcb-9c9f-e07111b74671\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp"
Jul 29 10:30:30.038600 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:30.038555    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4d62d964-dccd-4dcb-9c9f-e07111b74671-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-d88rp\" (UID: \"4d62d964-dccd-4dcb-9c9f-e07111b74671\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp"
Jul 29 10:30:30.038786 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:30.038611    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbhmm\" (UniqueName: \"kubernetes.io/projected/4d62d964-dccd-4dcb-9c9f-e07111b74671-kube-api-access-xbhmm\") pod \"monitoring-plugin-68b7fc9574-d88rp\" (UID: \"4d62d964-dccd-4dcb-9c9f-e07111b74671\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp"
Jul 29 10:30:30.038786 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:30:30.038741    2584 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 29 10:30:30.038887 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:30:30.038805    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d62d964-dccd-4dcb-9c9f-e07111b74671-monitoring-plugin-cert podName:4d62d964-dccd-4dcb-9c9f-e07111b74671 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:30.538790048 +0000 UTC m=+98.731817373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/4d62d964-dccd-4dcb-9c9f-e07111b74671-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-d88rp" (UID: "4d62d964-dccd-4dcb-9c9f-e07111b74671") : secret "monitoring-plugin-cert" not found
Jul 29 10:30:30.048079 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:30.048054    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbhmm\" (UniqueName: \"kubernetes.io/projected/4d62d964-dccd-4dcb-9c9f-e07111b74671-kube-api-access-xbhmm\") pod \"monitoring-plugin-68b7fc9574-d88rp\" (UID: \"4d62d964-dccd-4dcb-9c9f-e07111b74671\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp"
Jul 29 10:30:30.542238 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:30.542212    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4d62d964-dccd-4dcb-9c9f-e07111b74671-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-d88rp\" (UID: \"4d62d964-dccd-4dcb-9c9f-e07111b74671\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp"
Jul 29 10:30:30.542378 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:30:30.542357    2584 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 29 10:30:30.542438 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:30:30.542419    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d62d964-dccd-4dcb-9c9f-e07111b74671-monitoring-plugin-cert podName:4d62d964-dccd-4dcb-9c9f-e07111b74671 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:31.542402691 +0000 UTC m=+99.735430015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/4d62d964-dccd-4dcb-9c9f-e07111b74671-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-d88rp" (UID: "4d62d964-dccd-4dcb-9c9f-e07111b74671") : secret "monitoring-plugin-cert" not found
Jul 29 10:30:30.702153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:30.702121    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"040367c1-1b23-442f-b75d-9df47bb63a34","Type":"ContainerStarted","Data":"464a2773248b6d02dddf9e80eb510e3f24ae6b29daaa4a7c4784b80c248f08fd"}
Jul 29 10:30:30.702309 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:30.702187    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"040367c1-1b23-442f-b75d-9df47bb63a34","Type":"ContainerStarted","Data":"fdae215dcf7bfaf12cba8b42e5a6aae9c77db634f78833affddb73d3e10c2efd"}
Jul 29 10:30:30.702309 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:30.702212    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"040367c1-1b23-442f-b75d-9df47bb63a34","Type":"ContainerStarted","Data":"4176d4af43f29b2dbde1692210e91ee081c8d47a4d6c1477c1c80c6453992b1e"}
Jul 29 10:30:30.702309 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:30.702237    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"040367c1-1b23-442f-b75d-9df47bb63a34","Type":"ContainerStarted","Data":"d2dbf88d75d263bc9174fedbe713e3ef5bde6fe1c76ba8f6b26192dfbf8bbf54"}
Jul 29 10:30:30.702309 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:30.702259    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"040367c1-1b23-442f-b75d-9df47bb63a34","Type":"ContainerStarted","Data":"96296aa2e2204a3c5e246c5d94101bfca1c7f7c46f6c96af3e717cf7c247112f"}
Jul 29 10:30:31.324748 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.322834    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 10:30:31.328615 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.328597    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.331911 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.331888    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 10:30:31.332238 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.332220    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 10:30:31.332308 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.332284    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 10:30:31.332368 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.332317    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 10:30:31.332417 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.332385    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-82hnq\""
Jul 29 10:30:31.332578 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.332563    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 10:30:31.333151 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.333136    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 10:30:31.333539 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.333519    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 10:30:31.333612 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.333521    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 10:30:31.333612 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.333526    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 10:30:31.333830 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.333816    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 10:30:31.333906 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.333844    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 10:30:31.333906 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.333898    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 10:30:31.334325 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.334310    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e6997m9mvmb6b\""
Jul 29 10:30:31.334325 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.334318    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 10:30:31.334452 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.334412    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 10:30:31.336343 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.336321    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 10:30:31.343770 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.343749    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 10:30:31.348816 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.348797    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.348902 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.348824    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.348902 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.348851    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.348902 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.348876    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.348902 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.348892    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.349062 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.348957    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk26s\" (UniqueName: \"kubernetes.io/projected/c08b1a86-03e5-4374-9ab9-33d295ad57fc-kube-api-access-mk26s\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.349062 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.349000    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.349062 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.349023    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.349062 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.349042    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.349202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.349097    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.349202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.349118    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-config\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.349202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.349140    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.349202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.349169    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.349202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.349186    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.349343 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.349206    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-web-config\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.349343 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.349245    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.349343 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.349261    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.349343 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.349279    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c08b1a86-03e5-4374-9ab9-33d295ad57fc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.349343 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.349331    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c08b1a86-03e5-4374-9ab9-33d295ad57fc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.349483 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.349347    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c08b1a86-03e5-4374-9ab9-33d295ad57fc-config-out\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450305 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450273    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450464 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450313    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450464 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450341    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450464 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450364    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk26s\" (UniqueName: \"kubernetes.io/projected/c08b1a86-03e5-4374-9ab9-33d295ad57fc-kube-api-access-mk26s\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450464 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450385    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450464 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450410    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450464 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450432    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450464 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450465    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450869 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450489    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-config\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450869 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450516    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450869 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450537    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450869 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450560    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450869 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450599    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-web-config\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450869 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450641    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450869 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450665    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450869 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450713    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c08b1a86-03e5-4374-9ab9-33d295ad57fc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450869 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450766    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c08b1a86-03e5-4374-9ab9-33d295ad57fc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450869 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450788    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c08b1a86-03e5-4374-9ab9-33d295ad57fc-config-out\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450869 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.450869 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.450854    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.451413 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.451267    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.451413 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.451309    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.451622 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.451595    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.452456 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.452129    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.452550 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.452499    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.453527 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.453500    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.454040 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.454014    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.454344 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.454319    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c08b1a86-03e5-4374-9ab9-33d295ad57fc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.454656 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.454632    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.455273 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.454875    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c08b1a86-03e5-4374-9ab9-33d295ad57fc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.455273 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.455206    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c08b1a86-03e5-4374-9ab9-33d295ad57fc-config-out\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.455273 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.455208    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.455469 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.455373    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.455528 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.455508    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.456196 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.456175    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.456274 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.456175    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-config\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.456274 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.456245    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-web-config\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.456409 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.456393    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c08b1a86-03e5-4374-9ab9-33d295ad57fc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.457321 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.457304    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c08b1a86-03e5-4374-9ab9-33d295ad57fc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.467552 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.467521    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk26s\" (UniqueName: \"kubernetes.io/projected/c08b1a86-03e5-4374-9ab9-33d295ad57fc-kube-api-access-mk26s\") pod \"prometheus-k8s-0\" (UID: \"c08b1a86-03e5-4374-9ab9-33d295ad57fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.551791 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.551753    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4d62d964-dccd-4dcb-9c9f-e07111b74671-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-d88rp\" (UID: \"4d62d964-dccd-4dcb-9c9f-e07111b74671\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp"
Jul 29 10:30:31.554106 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.554088    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4d62d964-dccd-4dcb-9c9f-e07111b74671-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-d88rp\" (UID: \"4d62d964-dccd-4dcb-9c9f-e07111b74671\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp"
Jul 29 10:30:31.639488 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.639410    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:31.700138 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.699972    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp"
Jul 29 10:30:31.712542 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.712505    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"040367c1-1b23-442f-b75d-9df47bb63a34","Type":"ContainerStarted","Data":"419fd3212d187c4ac99788d2b84f798d06af7f9038c88fd1cf01e265a8283ac8"}
Jul 29 10:30:31.745764 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.745708    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.083445508 podStartE2EDuration="5.74567198s" totalImagesPullingTime="3.646986258s" totalInitContainerRuntime="15.240214ms" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:31.745360096 +0000 UTC m=+99.938387451" watchObservedRunningTime="2026-07-29 10:30:31.74567198 +0000 UTC m=+99.938699390"
Jul 29 10:30:31.796787 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.796758    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 10:30:31.800795 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:30:31.800763    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08b1a86_03e5_4374_9ab9_33d295ad57fc.slice/crio-3816a661422a373eb93875cab366ba5445f383c926aa3048ba5e4e1971500f8d WatchSource:0}: Error finding container 3816a661422a373eb93875cab366ba5445f383c926aa3048ba5e4e1971500f8d: Status 404 returned error can't find the container with id 3816a661422a373eb93875cab366ba5445f383c926aa3048ba5e4e1971500f8d
Jul 29 10:30:31.843131 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:31.843104    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp"]
Jul 29 10:30:31.846022 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:30:31.845997    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d62d964_dccd_4dcb_9c9f_e07111b74671.slice/crio-2915f67eb65b9bdad1b7895492bd93ed3d99ca0ea52de924eb795aa46b9d4f80 WatchSource:0}: Error finding container 2915f67eb65b9bdad1b7895492bd93ed3d99ca0ea52de924eb795aa46b9d4f80: Status 404 returned error can't find the container with id 2915f67eb65b9bdad1b7895492bd93ed3d99ca0ea52de924eb795aa46b9d4f80
Jul 29 10:30:32.719400 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:32.719367    2584 generic.go:356] "Generic (PLEG): container finished" podID="c08b1a86-03e5-4374-9ab9-33d295ad57fc" containerID="6e8bc4889a07781fe9940473a5c63848f156f32dbe694c91713e3081cf3ee181" exitCode=0
Jul 29 10:30:32.719803 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:32.719448    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c08b1a86-03e5-4374-9ab9-33d295ad57fc","Type":"ContainerDied","Data":"6e8bc4889a07781fe9940473a5c63848f156f32dbe694c91713e3081cf3ee181"}
Jul 29 10:30:32.719803 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:32.719563    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c08b1a86-03e5-4374-9ab9-33d295ad57fc","Type":"ContainerStarted","Data":"3816a661422a373eb93875cab366ba5445f383c926aa3048ba5e4e1971500f8d"}
Jul 29 10:30:32.720828 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:32.720737    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp" event={"ID":"4d62d964-dccd-4dcb-9c9f-e07111b74671","Type":"ContainerStarted","Data":"2915f67eb65b9bdad1b7895492bd93ed3d99ca0ea52de924eb795aa46b9d4f80"}
Jul 29 10:30:33.727711 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:33.727666    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp" event={"ID":"4d62d964-dccd-4dcb-9c9f-e07111b74671","Type":"ContainerStarted","Data":"3bc536cf8fdd6ec7b8d2c0a9c26e688a9bffa17cda65d1b770849fff77239c02"}
Jul 29 10:30:33.728158 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:33.728015    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp"
Jul 29 10:30:33.733562 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:33.733536    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp"
Jul 29 10:30:33.745881 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:33.745832    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-d88rp" podStartSLOduration=3.489206699 podStartE2EDuration="4.745817513s" totalImagesPullingTime="1.256610814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:33.744890839 +0000 UTC m=+101.937918191" watchObservedRunningTime="2026-07-29 10:30:33.745817513 +0000 UTC m=+101.938844850"
Jul 29 10:30:34.581480 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:34.581444    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fmxtb"
Jul 29 10:30:36.745790 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:36.745712    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c08b1a86-03e5-4374-9ab9-33d295ad57fc","Type":"ContainerStarted","Data":"98fabaa9dc2f17dd52d247cf99d0e708df65d2170b7a6af6775681213654e936"}
Jul 29 10:30:36.745790 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:36.745783    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c08b1a86-03e5-4374-9ab9-33d295ad57fc","Type":"ContainerStarted","Data":"5112b8de9fb05abe1b47aa21497a15963fab9762888a6e079d3108f3e79597d0"}
Jul 29 10:30:39.759641 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:39.759607    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c08b1a86-03e5-4374-9ab9-33d295ad57fc","Type":"ContainerStarted","Data":"e2fb7054ddf3d34e0a11b1859f7efd51520906163601d6fce498bb8d72259a56"}
Jul 29 10:30:39.760052 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:39.759713    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c08b1a86-03e5-4374-9ab9-33d295ad57fc","Type":"ContainerStarted","Data":"f934e97214df5055cc6f1c4fb6ed4da9eecc6efb62482a2d11e080de687573b5"}
Jul 29 10:30:39.760052 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:39.759744    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c08b1a86-03e5-4374-9ab9-33d295ad57fc","Type":"ContainerStarted","Data":"71f8dc6099ef5ffbe5ed88d302df915d352d4bbf1c36973be4dad62d8324537f"}
Jul 29 10:30:39.760052 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:39.759767    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c08b1a86-03e5-4374-9ab9-33d295ad57fc","Type":"ContainerStarted","Data":"6b9705990890e5905a9a6e8f8b4de7475eae6a5b5326ed0e81caa168d6761f7f"}
Jul 29 10:30:39.788006 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:39.787947    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.100235303 podStartE2EDuration="8.787925522s" totalImagesPullingTime="5.668639108s" totalInitContainerRuntime="19.051111ms" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:39.786103941 +0000 UTC m=+107.979131288" watchObservedRunningTime="2026-07-29 10:30:39.787925522 +0000 UTC m=+107.980952870"
Jul 29 10:30:41.639569 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:41.639533    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:51.806352 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:51.806321    2584 generic.go:356] "Generic (PLEG): container finished" podID="8fa58554-9c9a-445a-8e67-b6a0425565f5" containerID="b9c756ebd1b5a7884f61fc309fadb86a2764819d33a4046f99b5c6d7ede0cb60" exitCode=0
Jul 29 10:30:51.806797 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:51.806394    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx" event={"ID":"8fa58554-9c9a-445a-8e67-b6a0425565f5","Type":"ContainerDied","Data":"b9c756ebd1b5a7884f61fc309fadb86a2764819d33a4046f99b5c6d7ede0cb60"}
Jul 29 10:30:51.806797 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:51.806748    2584 scope.go:117] "RemoveContainer" containerID="b9c756ebd1b5a7884f61fc309fadb86a2764819d33a4046f99b5c6d7ede0cb60"
Jul 29 10:30:52.812359 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:30:52.812323    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-b7nxx" event={"ID":"8fa58554-9c9a-445a-8e67-b6a0425565f5","Type":"ContainerStarted","Data":"345aa1a183759b1612131b66c5b8e3183efb1cbef47d64ccb221bb19d939c024"}
Jul 29 10:31:31.640072 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:31:31.640042    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:31:31.659198 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:31:31.659174    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:31:31.978975 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:31:31.978897    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:33:52.161600 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:33:52.161574    2584 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 10:36:31.100156 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:31.100120    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dlsxd"]
Jul 29 10:36:31.103321 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:31.103294    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dlsxd"
Jul 29 10:36:31.106329 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:31.106308    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 29 10:36:31.106469 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:31.106379    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 29 10:36:31.107726 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:31.107709    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-f2q4w\""
Jul 29 10:36:31.112238 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:31.112215    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dlsxd"]
Jul 29 10:36:31.264935 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:31.264898    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-679nq\" (UniqueName: \"kubernetes.io/projected/c566d50b-1286-49d2-8f70-af508fd849c2-kube-api-access-679nq\") pod \"cert-manager-webhook-6486ffd997-dlsxd\" (UID: \"c566d50b-1286-49d2-8f70-af508fd849c2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dlsxd"
Jul 29 10:36:31.265111 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:31.264958    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c566d50b-1286-49d2-8f70-af508fd849c2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dlsxd\" (UID: \"c566d50b-1286-49d2-8f70-af508fd849c2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dlsxd"
Jul 29 10:36:31.366272 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:31.366186    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c566d50b-1286-49d2-8f70-af508fd849c2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dlsxd\" (UID: \"c566d50b-1286-49d2-8f70-af508fd849c2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dlsxd"
Jul 29 10:36:31.366272 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:31.366261    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-679nq\" (UniqueName: \"kubernetes.io/projected/c566d50b-1286-49d2-8f70-af508fd849c2-kube-api-access-679nq\") pod \"cert-manager-webhook-6486ffd997-dlsxd\" (UID: \"c566d50b-1286-49d2-8f70-af508fd849c2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dlsxd"
Jul 29 10:36:31.374993 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:31.374959    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c566d50b-1286-49d2-8f70-af508fd849c2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dlsxd\" (UID: \"c566d50b-1286-49d2-8f70-af508fd849c2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dlsxd"
Jul 29 10:36:31.375128 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:31.375037    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-679nq\" (UniqueName: \"kubernetes.io/projected/c566d50b-1286-49d2-8f70-af508fd849c2-kube-api-access-679nq\") pod \"cert-manager-webhook-6486ffd997-dlsxd\" (UID: \"c566d50b-1286-49d2-8f70-af508fd849c2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dlsxd"
Jul 29 10:36:31.412056 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:31.412024    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dlsxd"
Jul 29 10:36:31.539737 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:31.539714    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dlsxd"]
Jul 29 10:36:31.542153 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:36:31.542126    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc566d50b_1286_49d2_8f70_af508fd849c2.slice/crio-350bd9f8e9029017557699acc02cb41cac4c03c88838619f714ab21ab9b5966f WatchSource:0}: Error finding container 350bd9f8e9029017557699acc02cb41cac4c03c88838619f714ab21ab9b5966f: Status 404 returned error can't find the container with id 350bd9f8e9029017557699acc02cb41cac4c03c88838619f714ab21ab9b5966f
Jul 29 10:36:31.544540 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:31.544522    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:36:32.073546 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:32.073511    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dlsxd" event={"ID":"c566d50b-1286-49d2-8f70-af508fd849c2","Type":"ContainerStarted","Data":"350bd9f8e9029017557699acc02cb41cac4c03c88838619f714ab21ab9b5966f"}
Jul 29 10:36:38.098041 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:38.098008    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dlsxd" event={"ID":"c566d50b-1286-49d2-8f70-af508fd849c2","Type":"ContainerStarted","Data":"1484a36056c6c464c8885dd78642bc90bbe5d4077d8a207472b0171758a30585"}
Jul 29 10:36:38.098453 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:38.098122    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dlsxd"
Jul 29 10:36:38.118192 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:38.118150    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-dlsxd" podStartSLOduration=0.990949052 podStartE2EDuration="7.118137605s" totalImagesPullingTime="6.127188553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:36:38.117353314 +0000 UTC m=+466.310380661" watchObservedRunningTime="2026-07-29 10:36:38.118137605 +0000 UTC m=+466.311164953"
Jul 29 10:36:44.104378 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:44.104347    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dlsxd"
Jul 29 10:36:56.619201 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.619170    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"]
Jul 29 10:36:56.622255 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.622238    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:36:56.625434 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.625406    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 10:36:56.625560 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.625501    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 10:36:56.626770 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.626738    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 10:36:56.626770 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.626768    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 10:36:56.626938 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.626768    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:36:56.626938 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.626762    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6tm4x\""
Jul 29 10:36:56.639002 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.638977    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"]
Jul 29 10:36:56.680715 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.680665    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba8b9c99-52d9-4b2a-8900-4a61dd0fe286-metrics-cert\") pod \"lws-controller-manager-5c847d9c7c-s7n7k\" (UID: \"ba8b9c99-52d9-4b2a-8900-4a61dd0fe286\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:36:56.680887 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.680762    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ba8b9c99-52d9-4b2a-8900-4a61dd0fe286-manager-config\") pod \"lws-controller-manager-5c847d9c7c-s7n7k\" (UID: \"ba8b9c99-52d9-4b2a-8900-4a61dd0fe286\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:36:56.680887 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.680808    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpzpk\" (UniqueName: \"kubernetes.io/projected/ba8b9c99-52d9-4b2a-8900-4a61dd0fe286-kube-api-access-kpzpk\") pod \"lws-controller-manager-5c847d9c7c-s7n7k\" (UID: \"ba8b9c99-52d9-4b2a-8900-4a61dd0fe286\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:36:56.680887 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.680843    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba8b9c99-52d9-4b2a-8900-4a61dd0fe286-cert\") pod \"lws-controller-manager-5c847d9c7c-s7n7k\" (UID: \"ba8b9c99-52d9-4b2a-8900-4a61dd0fe286\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:36:56.781735 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.781659    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba8b9c99-52d9-4b2a-8900-4a61dd0fe286-cert\") pod \"lws-controller-manager-5c847d9c7c-s7n7k\" (UID: \"ba8b9c99-52d9-4b2a-8900-4a61dd0fe286\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:36:56.781929 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.781767    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba8b9c99-52d9-4b2a-8900-4a61dd0fe286-metrics-cert\") pod \"lws-controller-manager-5c847d9c7c-s7n7k\" (UID: \"ba8b9c99-52d9-4b2a-8900-4a61dd0fe286\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:36:56.781929 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.781798    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ba8b9c99-52d9-4b2a-8900-4a61dd0fe286-manager-config\") pod \"lws-controller-manager-5c847d9c7c-s7n7k\" (UID: \"ba8b9c99-52d9-4b2a-8900-4a61dd0fe286\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:36:56.781929 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.781834    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpzpk\" (UniqueName: \"kubernetes.io/projected/ba8b9c99-52d9-4b2a-8900-4a61dd0fe286-kube-api-access-kpzpk\") pod \"lws-controller-manager-5c847d9c7c-s7n7k\" (UID: \"ba8b9c99-52d9-4b2a-8900-4a61dd0fe286\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:36:56.782413 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.782391    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ba8b9c99-52d9-4b2a-8900-4a61dd0fe286-manager-config\") pod \"lws-controller-manager-5c847d9c7c-s7n7k\" (UID: \"ba8b9c99-52d9-4b2a-8900-4a61dd0fe286\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:36:56.784100 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.784078    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba8b9c99-52d9-4b2a-8900-4a61dd0fe286-metrics-cert\") pod \"lws-controller-manager-5c847d9c7c-s7n7k\" (UID: \"ba8b9c99-52d9-4b2a-8900-4a61dd0fe286\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:36:56.784223 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.784205    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba8b9c99-52d9-4b2a-8900-4a61dd0fe286-cert\") pod \"lws-controller-manager-5c847d9c7c-s7n7k\" (UID: \"ba8b9c99-52d9-4b2a-8900-4a61dd0fe286\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:36:56.791220 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.791195    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpzpk\" (UniqueName: \"kubernetes.io/projected/ba8b9c99-52d9-4b2a-8900-4a61dd0fe286-kube-api-access-kpzpk\") pod \"lws-controller-manager-5c847d9c7c-s7n7k\" (UID: \"ba8b9c99-52d9-4b2a-8900-4a61dd0fe286\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:36:56.931265 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:56.931176    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:36:57.097193 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:57.097141    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"]
Jul 29 10:36:57.099644 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:36:57.099618    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8b9c99_52d9_4b2a_8900_4a61dd0fe286.slice/crio-ae660934da6716d1f3464a75ef71f3e5440b2a43e61506a84228efe8ff8c85d5 WatchSource:0}: Error finding container ae660934da6716d1f3464a75ef71f3e5440b2a43e61506a84228efe8ff8c85d5: Status 404 returned error can't find the container with id ae660934da6716d1f3464a75ef71f3e5440b2a43e61506a84228efe8ff8c85d5
Jul 29 10:36:57.169339 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:36:57.169306    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k" event={"ID":"ba8b9c99-52d9-4b2a-8900-4a61dd0fe286","Type":"ContainerStarted","Data":"ae660934da6716d1f3464a75ef71f3e5440b2a43e61506a84228efe8ff8c85d5"}
Jul 29 10:37:03.193380 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.193339    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k" event={"ID":"ba8b9c99-52d9-4b2a-8900-4a61dd0fe286","Type":"ContainerStarted","Data":"a744ba9bb0b7e65604cfd8427bc70134562b2767ba6a1197319216bbac904dff"}
Jul 29 10:37:03.193773 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.193510    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:37:03.213021 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.212973    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k" podStartSLOduration=2.16583059 podStartE2EDuration="7.212958033s" totalImagesPullingTime="5.047127443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:37:03.211071891 +0000 UTC m=+491.404099238" watchObservedRunningTime="2026-07-29 10:37:03.212958033 +0000 UTC m=+491.405985379"
Jul 29 10:37:03.752748 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.752712    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"]
Jul 29 10:37:03.756419 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.756399    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.759562 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.759538    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 10:37:03.759720 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.759704    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 10:37:03.759906 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.759878    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 10:37:03.760020 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.760002    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-qbm9c\""
Jul 29 10:37:03.774586 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.774561    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"]
Jul 29 10:37:03.842360 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.842328    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.842360 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.842360    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.842595 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.842391    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.842595 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.842411    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.842595 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.842456    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.842595 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.842502    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.842595 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.842527    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzsq9\" (UniqueName: \"kubernetes.io/projected/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-kube-api-access-pzsq9\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.842595 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.842545    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.842831 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.842608    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.943411 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.943374    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.943411 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.943417    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.943628 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.943447    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.943628 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.943493    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.943628 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.943523    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzsq9\" (UniqueName: \"kubernetes.io/projected/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-kube-api-access-pzsq9\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.943628 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.943549    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.943628 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.943598    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.943900 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.943628    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.943900 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.943650    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.943998 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.943974    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.944052 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.944015    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.944267 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.944247    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.944359 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.944296    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.944359 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.944328    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.946232 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.946211    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.946461 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.946443    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.963965 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.963942    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzsq9\" (UniqueName: \"kubernetes.io/projected/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-kube-api-access-pzsq9\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:03.964216 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:03.964197    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b053698e-b8cf-4b7e-80d1-ed373c08cfb4-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-xf9js\" (UID: \"b053698e-b8cf-4b7e-80d1-ed373c08cfb4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:04.069416 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:04.069387    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:04.205709 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:04.205621    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"]
Jul 29 10:37:04.208297 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:37:04.208273    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb053698e_b8cf_4b7e_80d1_ed373c08cfb4.slice/crio-ce0788935c818c6278eca03058790b0d14af656f76646320cc100d0b17c1e27f WatchSource:0}: Error finding container ce0788935c818c6278eca03058790b0d14af656f76646320cc100d0b17c1e27f: Status 404 returned error can't find the container with id ce0788935c818c6278eca03058790b0d14af656f76646320cc100d0b17c1e27f
Jul 29 10:37:05.207689 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:05.207636    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js" event={"ID":"b053698e-b8cf-4b7e-80d1-ed373c08cfb4","Type":"ContainerStarted","Data":"ce0788935c818c6278eca03058790b0d14af656f76646320cc100d0b17c1e27f"}
Jul 29 10:37:06.757622 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:06.757583    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 10:37:06.757891 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:06.757659    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 10:37:06.757891 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:06.757705    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 10:37:07.217059 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:07.217023    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js" event={"ID":"b053698e-b8cf-4b7e-80d1-ed373c08cfb4","Type":"ContainerStarted","Data":"ed52526a8b3a73232d05987fd63797c65419870bdac099e360b92d4650e198d5"}
Jul 29 10:37:07.239200 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:07.239149    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js" podStartSLOduration=1.6919522470000001 podStartE2EDuration="4.239133479s" totalImagesPullingTime="2.547181232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:37:07.23645386 +0000 UTC m=+495.429481228" watchObservedRunningTime="2026-07-29 10:37:07.239133479 +0000 UTC m=+495.432160826"
Jul 29 10:37:08.070326 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:08.070283    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:08.074781 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:08.074758    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:08.221366 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:08.221337    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:08.222394 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:08.222377    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:37:14.199454 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:14.199426    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-s7n7k"
Jul 29 10:37:42.316251 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:42.316211    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-fdkhh"]
Jul 29 10:37:42.328546 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:42.328516    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-fdkhh"
Jul 29 10:37:42.330269 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:42.330243    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-fdkhh"]
Jul 29 10:37:42.331661 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:42.331642    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 10:37:42.331816 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:42.331722    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 10:37:42.331816 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:42.331643    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wzr2g\""
Jul 29 10:37:42.376117 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:42.376080    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkjsx\" (UniqueName: \"kubernetes.io/projected/9be44275-8cca-4b3e-ba47-3158d7e6be46-kube-api-access-fkjsx\") pod \"limitador-operator-controller-manager-774d94476f-fdkhh\" (UID: \"9be44275-8cca-4b3e-ba47-3158d7e6be46\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-fdkhh"
Jul 29 10:37:42.476887 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:42.476857    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjsx\" (UniqueName: \"kubernetes.io/projected/9be44275-8cca-4b3e-ba47-3158d7e6be46-kube-api-access-fkjsx\") pod \"limitador-operator-controller-manager-774d94476f-fdkhh\" (UID: \"9be44275-8cca-4b3e-ba47-3158d7e6be46\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-fdkhh"
Jul 29 10:37:42.485746 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:42.485724    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkjsx\" (UniqueName: \"kubernetes.io/projected/9be44275-8cca-4b3e-ba47-3158d7e6be46-kube-api-access-fkjsx\") pod \"limitador-operator-controller-manager-774d94476f-fdkhh\" (UID: \"9be44275-8cca-4b3e-ba47-3158d7e6be46\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-fdkhh"
Jul 29 10:37:42.641137 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:42.641057    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-fdkhh"
Jul 29 10:37:42.776331 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:42.776304    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-fdkhh"]
Jul 29 10:37:42.778481 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:37:42.778448    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be44275_8cca_4b3e_ba47_3158d7e6be46.slice/crio-303a4257df5a7130e9efac197ac62953c030c5b2fdc4845b86ef2a42c5dbc354 WatchSource:0}: Error finding container 303a4257df5a7130e9efac197ac62953c030c5b2fdc4845b86ef2a42c5dbc354: Status 404 returned error can't find the container with id 303a4257df5a7130e9efac197ac62953c030c5b2fdc4845b86ef2a42c5dbc354
Jul 29 10:37:43.360833 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:43.360798    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-fdkhh" event={"ID":"9be44275-8cca-4b3e-ba47-3158d7e6be46","Type":"ContainerStarted","Data":"303a4257df5a7130e9efac197ac62953c030c5b2fdc4845b86ef2a42c5dbc354"}
Jul 29 10:37:47.380453 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:47.380416    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-fdkhh" event={"ID":"9be44275-8cca-4b3e-ba47-3158d7e6be46","Type":"ContainerStarted","Data":"d5ff121190b794e155ce6ca2818aa1c0ef624a7ac245c0496215ae169133f494"}
Jul 29 10:37:47.380900 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:47.380503    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-fdkhh"
Jul 29 10:37:47.401566 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:47.401515    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-fdkhh" podStartSLOduration=1.154393918 podStartE2EDuration="5.401501182s" totalImagesPullingTime="4.247107264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:37:47.399075585 +0000 UTC m=+535.592102979" watchObservedRunningTime="2026-07-29 10:37:47.401501182 +0000 UTC m=+535.594528529"
Jul 29 10:37:58.386957 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:37:58.386931    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-fdkhh"
Jul 29 10:38:27.607216 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:27.607182    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-kn2s9"]
Jul 29 10:38:27.615165 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:27.615144    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-kn2s9"
Jul 29 10:38:27.617958 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:27.617924    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-kn2s9"]
Jul 29 10:38:27.618142 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:27.618125    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x8ngz\""
Jul 29 10:38:27.659870 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:27.659839    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqmvz\" (UniqueName: \"kubernetes.io/projected/c8105014-4849-4317-8ad0-761861028f62-kube-api-access-qqmvz\") pod \"authorino-b54df94b4-kn2s9\" (UID: \"c8105014-4849-4317-8ad0-761861028f62\") " pod="kuadrant-system/authorino-b54df94b4-kn2s9"
Jul 29 10:38:27.761213 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:27.761179    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqmvz\" (UniqueName: \"kubernetes.io/projected/c8105014-4849-4317-8ad0-761861028f62-kube-api-access-qqmvz\") pod \"authorino-b54df94b4-kn2s9\" (UID: \"c8105014-4849-4317-8ad0-761861028f62\") " pod="kuadrant-system/authorino-b54df94b4-kn2s9"
Jul 29 10:38:27.772812 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:27.772781    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqmvz\" (UniqueName: \"kubernetes.io/projected/c8105014-4849-4317-8ad0-761861028f62-kube-api-access-qqmvz\") pod \"authorino-b54df94b4-kn2s9\" (UID: \"c8105014-4849-4317-8ad0-761861028f62\") " pod="kuadrant-system/authorino-b54df94b4-kn2s9"
Jul 29 10:38:27.925534 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:27.925438    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-kn2s9"
Jul 29 10:38:28.053521 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:28.053500    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-kn2s9"]
Jul 29 10:38:28.056128 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:38:28.056095    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8105014_4849_4317_8ad0_761861028f62.slice/crio-764a18b327a9065f2729911404ded0a43a637f8cf41ecd2210ae703e87526f0b WatchSource:0}: Error finding container 764a18b327a9065f2729911404ded0a43a637f8cf41ecd2210ae703e87526f0b: Status 404 returned error can't find the container with id 764a18b327a9065f2729911404ded0a43a637f8cf41ecd2210ae703e87526f0b
Jul 29 10:38:28.542188 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:28.542155    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-kn2s9" event={"ID":"c8105014-4849-4317-8ad0-761861028f62","Type":"ContainerStarted","Data":"764a18b327a9065f2729911404ded0a43a637f8cf41ecd2210ae703e87526f0b"}
Jul 29 10:38:30.553319 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:30.553285    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-kn2s9" event={"ID":"c8105014-4849-4317-8ad0-761861028f62","Type":"ContainerStarted","Data":"ba6a871ce377600fab1b9a2ba189120e96e38a33d245b88d9e956c3c6f25df23"}
Jul 29 10:38:30.570562 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:30.570508    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-kn2s9" podStartSLOduration=1.486553399 podStartE2EDuration="3.570494362s" totalImagesPullingTime="2.083940963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:38:30.569044126 +0000 UTC m=+578.762071475" watchObservedRunningTime="2026-07-29 10:38:30.570494362 +0000 UTC m=+578.763521711"
Jul 29 10:38:31.133873 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:31.133840    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-kn2s9"]
Jul 29 10:38:32.561020 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:32.560980    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-kn2s9" podUID="c8105014-4849-4317-8ad0-761861028f62" containerName="authorino" containerID="cri-o://ba6a871ce377600fab1b9a2ba189120e96e38a33d245b88d9e956c3c6f25df23" gracePeriod=30
Jul 29 10:38:32.809739 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:32.809718    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-kn2s9"
Jul 29 10:38:32.905447 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:32.905363    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqmvz\" (UniqueName: \"kubernetes.io/projected/c8105014-4849-4317-8ad0-761861028f62-kube-api-access-qqmvz\") pod \"c8105014-4849-4317-8ad0-761861028f62\" (UID: \"c8105014-4849-4317-8ad0-761861028f62\") "
Jul 29 10:38:32.907519 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:32.907490    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8105014-4849-4317-8ad0-761861028f62-kube-api-access-qqmvz" (OuterVolumeSpecName: "kube-api-access-qqmvz") pod "c8105014-4849-4317-8ad0-761861028f62" (UID: "c8105014-4849-4317-8ad0-761861028f62"). InnerVolumeSpecName "kube-api-access-qqmvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:38:33.006198 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:33.006150    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqmvz\" (UniqueName: \"kubernetes.io/projected/c8105014-4849-4317-8ad0-761861028f62-kube-api-access-qqmvz\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:38:33.565980 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:33.565940    2584 generic.go:356] "Generic (PLEG): container finished" podID="c8105014-4849-4317-8ad0-761861028f62" containerID="ba6a871ce377600fab1b9a2ba189120e96e38a33d245b88d9e956c3c6f25df23" exitCode=0
Jul 29 10:38:33.565980 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:33.565986    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-kn2s9"
Jul 29 10:38:33.566427 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:33.566027    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-kn2s9" event={"ID":"c8105014-4849-4317-8ad0-761861028f62","Type":"ContainerDied","Data":"ba6a871ce377600fab1b9a2ba189120e96e38a33d245b88d9e956c3c6f25df23"}
Jul 29 10:38:33.566427 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:33.566068    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-kn2s9" event={"ID":"c8105014-4849-4317-8ad0-761861028f62","Type":"ContainerDied","Data":"764a18b327a9065f2729911404ded0a43a637f8cf41ecd2210ae703e87526f0b"}
Jul 29 10:38:33.566427 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:33.566086    2584 scope.go:117] "RemoveContainer" containerID="ba6a871ce377600fab1b9a2ba189120e96e38a33d245b88d9e956c3c6f25df23"
Jul 29 10:38:33.574366 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:33.574348    2584 scope.go:117] "RemoveContainer" containerID="ba6a871ce377600fab1b9a2ba189120e96e38a33d245b88d9e956c3c6f25df23"
Jul 29 10:38:33.574610 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:38:33.574589    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6a871ce377600fab1b9a2ba189120e96e38a33d245b88d9e956c3c6f25df23\": container with ID starting with ba6a871ce377600fab1b9a2ba189120e96e38a33d245b88d9e956c3c6f25df23 not found: ID does not exist" containerID="ba6a871ce377600fab1b9a2ba189120e96e38a33d245b88d9e956c3c6f25df23"
Jul 29 10:38:33.574695 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:33.574619    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6a871ce377600fab1b9a2ba189120e96e38a33d245b88d9e956c3c6f25df23"} err="failed to get container status \"ba6a871ce377600fab1b9a2ba189120e96e38a33d245b88d9e956c3c6f25df23\": rpc error: code = NotFound desc = could not find container \"ba6a871ce377600fab1b9a2ba189120e96e38a33d245b88d9e956c3c6f25df23\": container with ID starting with ba6a871ce377600fab1b9a2ba189120e96e38a33d245b88d9e956c3c6f25df23 not found: ID does not exist"
Jul 29 10:38:33.586140 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:33.586119    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-kn2s9"]
Jul 29 10:38:33.587951 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:33.587933    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-kn2s9"]
Jul 29 10:38:34.255431 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:34.255397    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8105014-4849-4317-8ad0-761861028f62" path="/var/lib/kubelet/pods/c8105014-4849-4317-8ad0-761861028f62/volumes"
Jul 29 10:38:58.927367 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:58.927337    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["observability/jaeger-6bbd7d4c9d-2rv7j"]
Jul 29 10:38:58.927871 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:58.927645    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8105014-4849-4317-8ad0-761861028f62" containerName="authorino"
Jul 29 10:38:58.927871 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:58.927657    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8105014-4849-4317-8ad0-761861028f62" containerName="authorino"
Jul 29 10:38:58.927871 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:58.927724    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8105014-4849-4317-8ad0-761861028f62" containerName="authorino"
Jul 29 10:38:58.931625 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:58.931610    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-2rv7j"
Jul 29 10:38:58.934586 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:58.934558    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"kube-root-ca.crt\""
Jul 29 10:38:58.934752 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:58.934601    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"openshift-service-ca.crt\""
Jul 29 10:38:58.935750 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:58.935729    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"observability\"/\"jaeger-dockercfg-kdmnk\""
Jul 29 10:38:58.944790 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:58.944769    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-2rv7j"]
Jul 29 10:38:59.032668 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:59.032631    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl4rj\" (UniqueName: \"kubernetes.io/projected/7e19d837-04e8-4db5-a12c-b18f2945c9cf-kube-api-access-pl4rj\") pod \"jaeger-6bbd7d4c9d-2rv7j\" (UID: \"7e19d837-04e8-4db5-a12c-b18f2945c9cf\") " pod="observability/jaeger-6bbd7d4c9d-2rv7j"
Jul 29 10:38:59.133047 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:59.133013    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl4rj\" (UniqueName: \"kubernetes.io/projected/7e19d837-04e8-4db5-a12c-b18f2945c9cf-kube-api-access-pl4rj\") pod \"jaeger-6bbd7d4c9d-2rv7j\" (UID: \"7e19d837-04e8-4db5-a12c-b18f2945c9cf\") " pod="observability/jaeger-6bbd7d4c9d-2rv7j"
Jul 29 10:38:59.143390 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:59.143360    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl4rj\" (UniqueName: \"kubernetes.io/projected/7e19d837-04e8-4db5-a12c-b18f2945c9cf-kube-api-access-pl4rj\") pod \"jaeger-6bbd7d4c9d-2rv7j\" (UID: \"7e19d837-04e8-4db5-a12c-b18f2945c9cf\") " pod="observability/jaeger-6bbd7d4c9d-2rv7j"
Jul 29 10:38:59.240477 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:59.240394    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-2rv7j"
Jul 29 10:38:59.368278 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:59.368256    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-2rv7j"]
Jul 29 10:38:59.370202 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:38:59.370169    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e19d837_04e8_4db5_a12c_b18f2945c9cf.slice/crio-25fc147d37ea517f19af452b9a81a79e8db030358f2b07f1b209501deaaabb07 WatchSource:0}: Error finding container 25fc147d37ea517f19af452b9a81a79e8db030358f2b07f1b209501deaaabb07: Status 404 returned error can't find the container with id 25fc147d37ea517f19af452b9a81a79e8db030358f2b07f1b209501deaaabb07
Jul 29 10:38:59.673602 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:38:59.673572    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-2rv7j" event={"ID":"7e19d837-04e8-4db5-a12c-b18f2945c9cf","Type":"ContainerStarted","Data":"25fc147d37ea517f19af452b9a81a79e8db030358f2b07f1b209501deaaabb07"}
Jul 29 10:39:01.686113 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:39:01.686081    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-2rv7j" event={"ID":"7e19d837-04e8-4db5-a12c-b18f2945c9cf","Type":"ContainerStarted","Data":"ee93333750407cdcf0e133fbe1f3f8c982d3bffa894f111dced3473f28cd3657"}
Jul 29 10:39:01.686526 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:39:01.686222    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="observability/jaeger-6bbd7d4c9d-2rv7j"
Jul 29 10:39:01.705792 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:39:01.705728    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="observability/jaeger-6bbd7d4c9d-2rv7j" podStartSLOduration=1.851838927 podStartE2EDuration="3.705710393s" totalImagesPullingTime="1.853871466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:39:01.703303547 +0000 UTC m=+609.896330906" watchObservedRunningTime="2026-07-29 10:39:01.705710393 +0000 UTC m=+609.898737740"
Jul 29 10:39:02.692541 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:39:02.692510    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="observability/jaeger-6bbd7d4c9d-2rv7j"
Jul 29 10:41:28.818792 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:28.818754    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"]
Jul 29 10:41:28.822132 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:28.822108    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:28.826457 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:28.826436    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 29 10:41:28.826457 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:28.826445    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 29 10:41:28.826632 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:28.826464    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-r8s7v\""
Jul 29 10:41:28.826632 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:28.826504    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 29 10:41:28.831415 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:28.831387    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"]
Jul 29 10:41:28.902314 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:28.902284    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-home\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:28.902459 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:28.902324    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/872eed29-67ca-4460-ae86-77f94f4e3e5f-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:28.902459 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:28.902344    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mpqp\" (UniqueName: \"kubernetes.io/projected/872eed29-67ca-4460-ae86-77f94f4e3e5f-kube-api-access-5mpqp\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:28.902459 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:28.902417    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-dshm\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:28.902459 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:28.902446    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:28.902605 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:28.902492    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:29.002897 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:29.002863    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:29.003065 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:29.002913    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-home\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:29.003065 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:29.002942    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/872eed29-67ca-4460-ae86-77f94f4e3e5f-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:29.003065 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:29.002957    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpqp\" (UniqueName: \"kubernetes.io/projected/872eed29-67ca-4460-ae86-77f94f4e3e5f-kube-api-access-5mpqp\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:29.003065 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:29.002987    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-dshm\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:29.003065 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:29.003007    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:29.003388 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:29.003356    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:29.003388 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:29.003367    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-home\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:29.003534 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:29.003403    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:29.005315 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:29.005287    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-dshm\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:29.005438 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:29.005420    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/872eed29-67ca-4460-ae86-77f94f4e3e5f-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:29.011877 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:29.011853    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mpqp\" (UniqueName: \"kubernetes.io/projected/872eed29-67ca-4460-ae86-77f94f4e3e5f-kube-api-access-5mpqp\") pod \"fc-smoke-test-kserve-7f4687897-s4nxc\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:29.133727 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:29.133624    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:29.266754 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:29.266717    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"]
Jul 29 10:41:29.269464 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:41:29.269436    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872eed29_67ca_4460_ae86_77f94f4e3e5f.slice/crio-37e997d0c859a28c9c0eafaaeab28fede8ca121ad4237c00d435f0ec69e5b21f WatchSource:0}: Error finding container 37e997d0c859a28c9c0eafaaeab28fede8ca121ad4237c00d435f0ec69e5b21f: Status 404 returned error can't find the container with id 37e997d0c859a28c9c0eafaaeab28fede8ca121ad4237c00d435f0ec69e5b21f
Jul 29 10:41:29.283788 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:29.283764    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc" event={"ID":"872eed29-67ca-4460-ae86-77f94f4e3e5f","Type":"ContainerStarted","Data":"37e997d0c859a28c9c0eafaaeab28fede8ca121ad4237c00d435f0ec69e5b21f"}
Jul 29 10:41:31.293000 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:31.292960    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc" event={"ID":"872eed29-67ca-4460-ae86-77f94f4e3e5f","Type":"ContainerStarted","Data":"40f62be63627fa905949b9f946a177ce1da1a37a188df44be5bc9c3e559cf47a"}
Jul 29 10:41:31.314049 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:31.314000    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc" podStartSLOduration=1.797465721 podStartE2EDuration="3.313983337s" totalImagesPullingTime="1.516517616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:41:31.313409873 +0000 UTC m=+759.506437233" watchObservedRunningTime="2026-07-29 10:41:31.313983337 +0000 UTC m=+759.507010696"
Jul 29 10:41:32.134577 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:32.134538    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:39.044771 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:39.044741    2584 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-xf9js container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": read tcp 10.134.0.2:39932->10.134.0.20:15021: read: connection reset by peer" start-of-body=
Jul 29 10:41:39.045189 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:39.044806    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js" podUID="b053698e-b8cf-4b7e-80d1-ed373c08cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": read tcp 10.134.0.2:39932->10.134.0.20:15021: read: connection reset by peer"
Jul 29 10:41:39.134161 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:39.134127    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:39.147359 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:39.147328    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:39.327026 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:39.326941    2584 generic.go:356] "Generic (PLEG): container finished" podID="b053698e-b8cf-4b7e-80d1-ed373c08cfb4" containerID="ed52526a8b3a73232d05987fd63797c65419870bdac099e360b92d4650e198d5" exitCode=0
Jul 29 10:41:39.327188 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:39.327019    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js" event={"ID":"b053698e-b8cf-4b7e-80d1-ed373c08cfb4","Type":"ContainerDied","Data":"ed52526a8b3a73232d05987fd63797c65419870bdac099e360b92d4650e198d5"}
Jul 29 10:41:39.327542 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:39.327506    2584 scope.go:117] "RemoveContainer" containerID="ed52526a8b3a73232d05987fd63797c65419870bdac099e360b92d4650e198d5"
Jul 29 10:41:39.328590 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:39.328557    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 10:41:39.328655 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:39.328639    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 10:41:39.328707 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:39.328688    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 10:41:39.339012 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:39.338989    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:41:40.346215 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:40.346174    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js" event={"ID":"b053698e-b8cf-4b7e-80d1-ed373c08cfb4","Type":"ContainerStarted","Data":"7991264069b0de8cdfb1900ee41a2a01e991432100b5306a53abf4557e63634b"}
Jul 29 10:41:41.070431 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:41.070394    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:41:41.070627 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:41.070601    2584 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-xf9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 29 10:41:41.070707 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:41.070659    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js" podUID="b053698e-b8cf-4b7e-80d1-ed373c08cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 29 10:41:42.070658 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:42.070620    2584 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-xf9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 29 10:41:42.071155 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:42.070702    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js" podUID="b053698e-b8cf-4b7e-80d1-ed373c08cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 29 10:41:43.070555 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:43.070519    2584 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-xf9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 29 10:41:43.070774 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:43.070572    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js" podUID="b053698e-b8cf-4b7e-80d1-ed373c08cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 29 10:41:44.070638 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:44.070601    2584 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-xf9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 29 10:41:44.070839 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:44.070655    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js" podUID="b053698e-b8cf-4b7e-80d1-ed373c08cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 29 10:41:45.069733 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:45.069697    2584 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-xf9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 29 10:41:45.069900 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:45.069756    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js" podUID="b053698e-b8cf-4b7e-80d1-ed373c08cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 29 10:41:46.771495 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:46.771464    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:41:46.772008 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:46.771777    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:41:46.772670 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:41:46.772655    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-xf9js"
Jul 29 10:42:09.484267 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.484233    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"]
Jul 29 10:42:09.493184 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:42:09.493140    2584 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:42:09.493184 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:42:09.493184 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:42:09.493184 ip-10-0-140-240 kubenswrapper[2584]:  > containerID="40f62be63627fa905949b9f946a177ce1da1a37a188df44be5bc9c3e559cf47a" cmd=["/bin/sleep","15"]
Jul 29 10:42:09.493464 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.493212    2584 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc" message="" err=<
Jul 29 10:42:09.493464 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:42:09.493464 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:42:09.493464 ip-10-0-140-240 kubenswrapper[2584]:  >
Jul 29 10:42:09.493464 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:42:09.493229    2584 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:42:09.493464 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:42:09.493464 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:42:09.493464 ip-10-0-140-240 kubenswrapper[2584]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc" podUID="872eed29-67ca-4460-ae86-77f94f4e3e5f" containerName="main" containerID="cri-o://40f62be63627fa905949b9f946a177ce1da1a37a188df44be5bc9c3e559cf47a"
Jul 29 10:42:09.493464 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.493284    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc" podUID="872eed29-67ca-4460-ae86-77f94f4e3e5f" containerName="main" containerID="cri-o://40f62be63627fa905949b9f946a177ce1da1a37a188df44be5bc9c3e559cf47a" gracePeriod=60
Jul 29 10:42:09.836354 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.836327    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:42:09.889374 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.889343    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-tmp-dir\") pod \"872eed29-67ca-4460-ae86-77f94f4e3e5f\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") "
Jul 29 10:42:09.889542 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.889384    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-dshm\") pod \"872eed29-67ca-4460-ae86-77f94f4e3e5f\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") "
Jul 29 10:42:09.889542 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.889409    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-model-cache\") pod \"872eed29-67ca-4460-ae86-77f94f4e3e5f\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") "
Jul 29 10:42:09.889542 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.889428    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/872eed29-67ca-4460-ae86-77f94f4e3e5f-tls-certs\") pod \"872eed29-67ca-4460-ae86-77f94f4e3e5f\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") "
Jul 29 10:42:09.889542 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.889470    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-home\") pod \"872eed29-67ca-4460-ae86-77f94f4e3e5f\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") "
Jul 29 10:42:09.889542 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.889490    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mpqp\" (UniqueName: \"kubernetes.io/projected/872eed29-67ca-4460-ae86-77f94f4e3e5f-kube-api-access-5mpqp\") pod \"872eed29-67ca-4460-ae86-77f94f4e3e5f\" (UID: \"872eed29-67ca-4460-ae86-77f94f4e3e5f\") "
Jul 29 10:42:09.889824 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.889667    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "872eed29-67ca-4460-ae86-77f94f4e3e5f" (UID: "872eed29-67ca-4460-ae86-77f94f4e3e5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:42:09.889824 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.889697    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "872eed29-67ca-4460-ae86-77f94f4e3e5f" (UID: "872eed29-67ca-4460-ae86-77f94f4e3e5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:42:09.889824 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.889801    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-home" (OuterVolumeSpecName: "home") pod "872eed29-67ca-4460-ae86-77f94f4e3e5f" (UID: "872eed29-67ca-4460-ae86-77f94f4e3e5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:42:09.889977 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.889885    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:42:09.889977 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.889907    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-model-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:42:09.889977 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.889920    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-home\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:42:09.891587 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.891556    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-dshm" (OuterVolumeSpecName: "dshm") pod "872eed29-67ca-4460-ae86-77f94f4e3e5f" (UID: "872eed29-67ca-4460-ae86-77f94f4e3e5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:42:09.891725 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.891637    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/872eed29-67ca-4460-ae86-77f94f4e3e5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "872eed29-67ca-4460-ae86-77f94f4e3e5f" (UID: "872eed29-67ca-4460-ae86-77f94f4e3e5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:42:09.891778 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.891726    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872eed29-67ca-4460-ae86-77f94f4e3e5f-kube-api-access-5mpqp" (OuterVolumeSpecName: "kube-api-access-5mpqp") pod "872eed29-67ca-4460-ae86-77f94f4e3e5f" (UID: "872eed29-67ca-4460-ae86-77f94f4e3e5f"). InnerVolumeSpecName "kube-api-access-5mpqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:42:09.991299 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.991264    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/872eed29-67ca-4460-ae86-77f94f4e3e5f-dshm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:42:09.991299 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.991292    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/872eed29-67ca-4460-ae86-77f94f4e3e5f-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:42:09.991299 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:09.991304    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mpqp\" (UniqueName: \"kubernetes.io/projected/872eed29-67ca-4460-ae86-77f94f4e3e5f-kube-api-access-5mpqp\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:42:10.476350 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:10.476315    2584 generic.go:356] "Generic (PLEG): container finished" podID="872eed29-67ca-4460-ae86-77f94f4e3e5f" containerID="40f62be63627fa905949b9f946a177ce1da1a37a188df44be5bc9c3e559cf47a" exitCode=0
Jul 29 10:42:10.476517 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:10.476366    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"
Jul 29 10:42:10.476517 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:10.476365    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc" event={"ID":"872eed29-67ca-4460-ae86-77f94f4e3e5f","Type":"ContainerDied","Data":"40f62be63627fa905949b9f946a177ce1da1a37a188df44be5bc9c3e559cf47a"}
Jul 29 10:42:10.476517 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:10.476473    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc" event={"ID":"872eed29-67ca-4460-ae86-77f94f4e3e5f","Type":"ContainerDied","Data":"37e997d0c859a28c9c0eafaaeab28fede8ca121ad4237c00d435f0ec69e5b21f"}
Jul 29 10:42:10.476517 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:10.476500    2584 scope.go:117] "RemoveContainer" containerID="40f62be63627fa905949b9f946a177ce1da1a37a188df44be5bc9c3e559cf47a"
Jul 29 10:42:10.485181 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:10.485162    2584 scope.go:117] "RemoveContainer" containerID="40f62be63627fa905949b9f946a177ce1da1a37a188df44be5bc9c3e559cf47a"
Jul 29 10:42:10.485473 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:42:10.485436    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f62be63627fa905949b9f946a177ce1da1a37a188df44be5bc9c3e559cf47a\": container with ID starting with 40f62be63627fa905949b9f946a177ce1da1a37a188df44be5bc9c3e559cf47a not found: ID does not exist" containerID="40f62be63627fa905949b9f946a177ce1da1a37a188df44be5bc9c3e559cf47a"
Jul 29 10:42:10.485473 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:10.485460    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f62be63627fa905949b9f946a177ce1da1a37a188df44be5bc9c3e559cf47a"} err="failed to get container status \"40f62be63627fa905949b9f946a177ce1da1a37a188df44be5bc9c3e559cf47a\": rpc error: code = NotFound desc = could not find container \"40f62be63627fa905949b9f946a177ce1da1a37a188df44be5bc9c3e559cf47a\": container with ID starting with 40f62be63627fa905949b9f946a177ce1da1a37a188df44be5bc9c3e559cf47a not found: ID does not exist"
Jul 29 10:42:10.493146 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:10.493123    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"]
Jul 29 10:42:10.497304 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:10.497283    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-s4nxc"]
Jul 29 10:42:12.254798 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:12.254763    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="872eed29-67ca-4460-ae86-77f94f4e3e5f" path="/var/lib/kubelet/pods/872eed29-67ca-4460-ae86-77f94f4e3e5f/volumes"
Jul 29 10:42:42.132841 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.132802    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"]
Jul 29 10:42:42.133326 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.133126    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872eed29-67ca-4460-ae86-77f94f4e3e5f" containerName="main"
Jul 29 10:42:42.133326 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.133137    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="872eed29-67ca-4460-ae86-77f94f4e3e5f" containerName="main"
Jul 29 10:42:42.133326 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.133209    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="872eed29-67ca-4460-ae86-77f94f4e3e5f" containerName="main"
Jul 29 10:42:42.136292 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.136269    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.139268 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.139239    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"]
Jul 29 10:42:42.140521 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.140488    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 29 10:42:42.140888 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.140867    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-klv5f\""
Jul 29 10:42:42.140997 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.140975    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 29 10:42:42.141098 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.141062    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 29 10:42:42.267015 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.266988    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.267159 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.267027    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-home\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.267159 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.267044    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/034f18c0-ed59-40c7-b6b2-c90d5144e290-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.267159 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.267091    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g2vr\" (UniqueName: \"kubernetes.io/projected/034f18c0-ed59-40c7-b6b2-c90d5144e290-kube-api-access-5g2vr\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.267159 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.267125    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.267159 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.267151    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.368382 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.368351    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g2vr\" (UniqueName: \"kubernetes.io/projected/034f18c0-ed59-40c7-b6b2-c90d5144e290-kube-api-access-5g2vr\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.368579 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.368413    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.368579 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.368443    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.368579 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.368565    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.368800 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.368643    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-home\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.368800 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.368700    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/034f18c0-ed59-40c7-b6b2-c90d5144e290-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.369005 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.368981    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.369155 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.369106    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-home\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.369275 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.369151    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.370733 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.370715    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.371061 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.371043    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/034f18c0-ed59-40c7-b6b2-c90d5144e290-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.378940 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.378920    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g2vr\" (UniqueName: \"kubernetes.io/projected/034f18c0-ed59-40c7-b6b2-c90d5144e290-kube-api-access-5g2vr\") pod \"fc-concurrency-test-kserve-b5d44467c-m4lzm\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.447865 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.447768    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:42.578697 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.578657    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"]
Jul 29 10:42:42.581076 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:42:42.581046    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod034f18c0_ed59_40c7_b6b2_c90d5144e290.slice/crio-d5455c738c2ad9a6c3e8ca5f6145cb94f305d7cacae64fea279e74b673955aba WatchSource:0}: Error finding container d5455c738c2ad9a6c3e8ca5f6145cb94f305d7cacae64fea279e74b673955aba: Status 404 returned error can't find the container with id d5455c738c2ad9a6c3e8ca5f6145cb94f305d7cacae64fea279e74b673955aba
Jul 29 10:42:42.609020 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:42.608996    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm" event={"ID":"034f18c0-ed59-40c7-b6b2-c90d5144e290","Type":"ContainerStarted","Data":"d5455c738c2ad9a6c3e8ca5f6145cb94f305d7cacae64fea279e74b673955aba"}
Jul 29 10:42:43.614705 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:43.614658    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm" event={"ID":"034f18c0-ed59-40c7-b6b2-c90d5144e290","Type":"ContainerStarted","Data":"b25c02cfee874cc6fe94c62d1205177b8cd1dad4ea12f280e7113e0aead0aada"}
Jul 29 10:42:43.632719 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:43.632656    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm" podStartSLOduration=1.6326429999999998 podStartE2EDuration="1.632643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:42:43.631025588 +0000 UTC m=+831.824052934" watchObservedRunningTime="2026-07-29 10:42:43.632643 +0000 UTC m=+831.825670347"
Jul 29 10:42:44.447877 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:44.447843    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:52.447964 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:52.447935    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:52.460609 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:52.460586    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:42:52.662361 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:42:52.662332    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:43:21.364288 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.364256    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"]
Jul 29 10:43:21.373015 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:43:21.372978    2584 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:43:21.373015 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:43:21.373015 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:43:21.373015 ip-10-0-140-240 kubenswrapper[2584]:  > containerID="b25c02cfee874cc6fe94c62d1205177b8cd1dad4ea12f280e7113e0aead0aada" cmd=["/bin/sleep","15"]
Jul 29 10:43:21.373320 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.373040    2584 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm" message="" err=<
Jul 29 10:43:21.373320 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:43:21.373320 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:43:21.373320 ip-10-0-140-240 kubenswrapper[2584]:  >
Jul 29 10:43:21.373320 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:43:21.373056    2584 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:43:21.373320 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:43:21.373320 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:43:21.373320 ip-10-0-140-240 kubenswrapper[2584]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm" podUID="034f18c0-ed59-40c7-b6b2-c90d5144e290" containerName="main" containerID="cri-o://b25c02cfee874cc6fe94c62d1205177b8cd1dad4ea12f280e7113e0aead0aada"
Jul 29 10:43:21.373320 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.373100    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm" podUID="034f18c0-ed59-40c7-b6b2-c90d5144e290" containerName="main" containerID="cri-o://b25c02cfee874cc6fe94c62d1205177b8cd1dad4ea12f280e7113e0aead0aada" gracePeriod=60
Jul 29 10:43:21.716552 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.716523    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:43:21.776937 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.776909    2584 generic.go:356] "Generic (PLEG): container finished" podID="034f18c0-ed59-40c7-b6b2-c90d5144e290" containerID="b25c02cfee874cc6fe94c62d1205177b8cd1dad4ea12f280e7113e0aead0aada" exitCode=0
Jul 29 10:43:21.777126 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.776949    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm" event={"ID":"034f18c0-ed59-40c7-b6b2-c90d5144e290","Type":"ContainerDied","Data":"b25c02cfee874cc6fe94c62d1205177b8cd1dad4ea12f280e7113e0aead0aada"}
Jul 29 10:43:21.777126 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.776957    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"
Jul 29 10:43:21.777126 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.776997    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm" event={"ID":"034f18c0-ed59-40c7-b6b2-c90d5144e290","Type":"ContainerDied","Data":"d5455c738c2ad9a6c3e8ca5f6145cb94f305d7cacae64fea279e74b673955aba"}
Jul 29 10:43:21.777126 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.777023    2584 scope.go:117] "RemoveContainer" containerID="b25c02cfee874cc6fe94c62d1205177b8cd1dad4ea12f280e7113e0aead0aada"
Jul 29 10:43:21.785382 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.785364    2584 scope.go:117] "RemoveContainer" containerID="b25c02cfee874cc6fe94c62d1205177b8cd1dad4ea12f280e7113e0aead0aada"
Jul 29 10:43:21.785642 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:43:21.785622    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25c02cfee874cc6fe94c62d1205177b8cd1dad4ea12f280e7113e0aead0aada\": container with ID starting with b25c02cfee874cc6fe94c62d1205177b8cd1dad4ea12f280e7113e0aead0aada not found: ID does not exist" containerID="b25c02cfee874cc6fe94c62d1205177b8cd1dad4ea12f280e7113e0aead0aada"
Jul 29 10:43:21.785707 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.785651    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25c02cfee874cc6fe94c62d1205177b8cd1dad4ea12f280e7113e0aead0aada"} err="failed to get container status \"b25c02cfee874cc6fe94c62d1205177b8cd1dad4ea12f280e7113e0aead0aada\": rpc error: code = NotFound desc = could not find container \"b25c02cfee874cc6fe94c62d1205177b8cd1dad4ea12f280e7113e0aead0aada\": container with ID starting with b25c02cfee874cc6fe94c62d1205177b8cd1dad4ea12f280e7113e0aead0aada not found: ID does not exist"
Jul 29 10:43:21.805955 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.805934    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/034f18c0-ed59-40c7-b6b2-c90d5144e290-tls-certs\") pod \"034f18c0-ed59-40c7-b6b2-c90d5144e290\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") "
Jul 29 10:43:21.806068 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.805965    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-dshm\") pod \"034f18c0-ed59-40c7-b6b2-c90d5144e290\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") "
Jul 29 10:43:21.806068 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.805984    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-home\") pod \"034f18c0-ed59-40c7-b6b2-c90d5144e290\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") "
Jul 29 10:43:21.806068 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.806034    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g2vr\" (UniqueName: \"kubernetes.io/projected/034f18c0-ed59-40c7-b6b2-c90d5144e290-kube-api-access-5g2vr\") pod \"034f18c0-ed59-40c7-b6b2-c90d5144e290\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") "
Jul 29 10:43:21.806217 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.806068    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-model-cache\") pod \"034f18c0-ed59-40c7-b6b2-c90d5144e290\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") "
Jul 29 10:43:21.806217 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.806095    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-tmp-dir\") pod \"034f18c0-ed59-40c7-b6b2-c90d5144e290\" (UID: \"034f18c0-ed59-40c7-b6b2-c90d5144e290\") "
Jul 29 10:43:21.806372 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.806334    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-home" (OuterVolumeSpecName: "home") pod "034f18c0-ed59-40c7-b6b2-c90d5144e290" (UID: "034f18c0-ed59-40c7-b6b2-c90d5144e290"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:43:21.806372 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.806349    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-model-cache" (OuterVolumeSpecName: "model-cache") pod "034f18c0-ed59-40c7-b6b2-c90d5144e290" (UID: "034f18c0-ed59-40c7-b6b2-c90d5144e290"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:43:21.806514 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.806444    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "034f18c0-ed59-40c7-b6b2-c90d5144e290" (UID: "034f18c0-ed59-40c7-b6b2-c90d5144e290"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:43:21.808202 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.808177    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-dshm" (OuterVolumeSpecName: "dshm") pod "034f18c0-ed59-40c7-b6b2-c90d5144e290" (UID: "034f18c0-ed59-40c7-b6b2-c90d5144e290"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:43:21.808278 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.808261    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/034f18c0-ed59-40c7-b6b2-c90d5144e290-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "034f18c0-ed59-40c7-b6b2-c90d5144e290" (UID: "034f18c0-ed59-40c7-b6b2-c90d5144e290"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:43:21.808278 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.808267    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034f18c0-ed59-40c7-b6b2-c90d5144e290-kube-api-access-5g2vr" (OuterVolumeSpecName: "kube-api-access-5g2vr") pod "034f18c0-ed59-40c7-b6b2-c90d5144e290" (UID: "034f18c0-ed59-40c7-b6b2-c90d5144e290"). InnerVolumeSpecName "kube-api-access-5g2vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:43:21.907090 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.907013    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:43:21.907090 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.907040    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/034f18c0-ed59-40c7-b6b2-c90d5144e290-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:43:21.907090 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.907048    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-dshm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:43:21.907090 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.907055    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-home\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:43:21.907090 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.907064    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g2vr\" (UniqueName: \"kubernetes.io/projected/034f18c0-ed59-40c7-b6b2-c90d5144e290-kube-api-access-5g2vr\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:43:21.907090 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:21.907073    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/034f18c0-ed59-40c7-b6b2-c90d5144e290-model-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:43:22.103625 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:22.103592    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"]
Jul 29 10:43:22.107346 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:22.107323    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm"]
Jul 29 10:43:22.254978 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:22.254900    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="034f18c0-ed59-40c7-b6b2-c90d5144e290" path="/var/lib/kubelet/pods/034f18c0-ed59-40c7-b6b2-c90d5144e290/volumes"
Jul 29 10:43:22.651819 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:22.651778    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-m4lzm" podUID="034f18c0-ed59-40c7-b6b2-c90d5144e290" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 29 10:43:41.603040 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.603008    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"]
Jul 29 10:43:41.603542 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.603522    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="034f18c0-ed59-40c7-b6b2-c90d5144e290" containerName="main"
Jul 29 10:43:41.603617 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.603545    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="034f18c0-ed59-40c7-b6b2-c90d5144e290" containerName="main"
Jul 29 10:43:41.603749 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.603643    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="034f18c0-ed59-40c7-b6b2-c90d5144e290" containerName="main"
Jul 29 10:43:41.606736 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.606716    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.609788 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.609764    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 29 10:43:41.609926 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.609764    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Jul 29 10:43:41.609992 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.609924    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 29 10:43:41.609992 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.609949    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-bh5ht\""
Jul 29 10:43:41.619222 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.619194    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"]
Jul 29 10:43:41.678187 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.678162    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.678322 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.678193    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.678322 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.678220    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.678322 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.678286    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxrx4\" (UniqueName: \"kubernetes.io/projected/a4551277-c244-420a-b4bf-c8ed35b3d9d5-kube-api-access-nxrx4\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.678449 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.678335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.678449 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.678385    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.678449 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.678403    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.678449 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.678424    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.678588 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.678549    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.778983 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.778952    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.779174 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.779011    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.779174 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.779042    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.779174 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.779062    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.779339 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.779193    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.779339 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.779248    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxrx4\" (UniqueName: \"kubernetes.io/projected/a4551277-c244-420a-b4bf-c8ed35b3d9d5-kube-api-access-nxrx4\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.779339 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.779289    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.779489 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.779358    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.779489 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.779384    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.779489 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.779445    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.779652 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.779507    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.779652 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.779554    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.779835 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.779758    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.779835 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.779762    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.781658 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.781630    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.781862 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.781845    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.787529 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.787505    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxrx4\" (UniqueName: \"kubernetes.io/projected/a4551277-c244-420a-b4bf-c8ed35b3d9d5-kube-api-access-nxrx4\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.787636 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.787567    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zqg95\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:41.938438 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:41.938353    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:42.072169 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:42.072146    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"]
Jul 29 10:43:42.074660 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:43:42.074631    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4551277_c244_420a_b4bf_c8ed35b3d9d5.slice/crio-aafbaf1ff5e7c34c5f83f5f26b1572cf31252e289577cee4db02e42bc115035d WatchSource:0}: Error finding container aafbaf1ff5e7c34c5f83f5f26b1572cf31252e289577cee4db02e42bc115035d: Status 404 returned error can't find the container with id aafbaf1ff5e7c34c5f83f5f26b1572cf31252e289577cee4db02e42bc115035d
Jul 29 10:43:42.076669 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:42.076639    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 10:43:42.076751 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:42.076729    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 10:43:42.076793 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:42.076771    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 10:43:42.870693 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:42.870647    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95" event={"ID":"a4551277-c244-420a-b4bf-c8ed35b3d9d5","Type":"ContainerStarted","Data":"c220f481c657fecf190b44ffc2eb33d6c0c16e1cd4f35295ad96d93e15eacfe1"}
Jul 29 10:43:42.871185 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:42.870710    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95" event={"ID":"a4551277-c244-420a-b4bf-c8ed35b3d9d5","Type":"ContainerStarted","Data":"aafbaf1ff5e7c34c5f83f5f26b1572cf31252e289577cee4db02e42bc115035d"}
Jul 29 10:43:42.917326 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:42.917273    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95" podStartSLOduration=1.917255896 podStartE2EDuration="1.917255896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:43:42.915450941 +0000 UTC m=+891.108478289" watchObservedRunningTime="2026-07-29 10:43:42.917255896 +0000 UTC m=+891.110283243"
Jul 29 10:43:42.939313 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:42.939277    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:43.943961 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:43.943883    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:44.880705 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:44.880637    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:44.881465 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:44.881444    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:43:51.405660 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.405620    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"]
Jul 29 10:43:51.409362 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.409340    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.413880 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.413861    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-99x4h\""
Jul 29 10:43:51.413997 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.413891    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 29 10:43:51.420257 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.420234    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"]
Jul 29 10:43:51.455995 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.455962    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.456126 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.456019    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.456126 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.456076    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832b0393-1d05-44ee-9da9-c81acee6ebec-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.456126 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.456109    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.456233 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.456128    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5tf5\" (UniqueName: \"kubernetes.io/projected/832b0393-1d05-44ee-9da9-c81acee6ebec-kube-api-access-h5tf5\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.456233 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.456180    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.556761 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.556719    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.556761 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.556765    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.556963 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.556804    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.556963 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.556831    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832b0393-1d05-44ee-9da9-c81acee6ebec-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.557036 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.556980    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.557036 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.557021    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5tf5\" (UniqueName: \"kubernetes.io/projected/832b0393-1d05-44ee-9da9-c81acee6ebec-kube-api-access-h5tf5\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.557179 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.557158    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.557248 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.557225    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.557334 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.557316    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.559122 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.559100    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.559365 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.559348    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832b0393-1d05-44ee-9da9-c81acee6ebec-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.565013 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.564991    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5tf5\" (UniqueName: \"kubernetes.io/projected/832b0393-1d05-44ee-9da9-c81acee6ebec-kube-api-access-h5tf5\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.723670 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.723591    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:51.853501 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.853478    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"]
Jul 29 10:43:51.855779 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:43:51.855750    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832b0393_1d05_44ee_9da9_c81acee6ebec.slice/crio-c84b857068a770364dc1148878084c929cd15922ae8e2caf1654e42060df3ce9 WatchSource:0}: Error finding container c84b857068a770364dc1148878084c929cd15922ae8e2caf1654e42060df3ce9: Status 404 returned error can't find the container with id c84b857068a770364dc1148878084c929cd15922ae8e2caf1654e42060df3ce9
Jul 29 10:43:51.914066 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.914040    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb" event={"ID":"832b0393-1d05-44ee-9da9-c81acee6ebec","Type":"ContainerStarted","Data":"e5955b38e586998b69bf134908f9d0dfb2aa5024117773b0e741d723a97b59f5"}
Jul 29 10:43:51.914164 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:51.914090    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb" event={"ID":"832b0393-1d05-44ee-9da9-c81acee6ebec","Type":"ContainerStarted","Data":"c84b857068a770364dc1148878084c929cd15922ae8e2caf1654e42060df3ce9"}
Jul 29 10:43:52.937609 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:52.937547    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb" podStartSLOduration=1.937526494 podStartE2EDuration="1.937526494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:43:52.935146739 +0000 UTC m=+901.128174087" watchObservedRunningTime="2026-07-29 10:43:52.937526494 +0000 UTC m=+901.130553842"
Jul 29 10:43:53.724074 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:53.724020    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:54.166181 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:54.166146    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"]
Jul 29 10:43:54.944699 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:43:54.944001    2584 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:43:54.944699 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:43:54.944699 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:43:54.944699 ip-10-0-140-240 kubenswrapper[2584]:  > containerID="e5955b38e586998b69bf134908f9d0dfb2aa5024117773b0e741d723a97b59f5" cmd=["/bin/sleep","15"]
Jul 29 10:43:54.944699 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:54.944069    2584 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb" message="" err=<
Jul 29 10:43:54.944699 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:43:54.944699 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:43:54.944699 ip-10-0-140-240 kubenswrapper[2584]:  >
Jul 29 10:43:54.944699 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:43:54.944087    2584 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:43:54.944699 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:43:54.944699 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:43:54.944699 ip-10-0-140-240 kubenswrapper[2584]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb" podUID="832b0393-1d05-44ee-9da9-c81acee6ebec" containerName="main" containerID="cri-o://e5955b38e586998b69bf134908f9d0dfb2aa5024117773b0e741d723a97b59f5"
Jul 29 10:43:54.944699 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:54.944140    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb" podUID="832b0393-1d05-44ee-9da9-c81acee6ebec" containerName="main" containerID="cri-o://e5955b38e586998b69bf134908f9d0dfb2aa5024117773b0e741d723a97b59f5" gracePeriod=60
Jul 29 10:43:55.207812 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.207753    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:55.292392 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.292365    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832b0393-1d05-44ee-9da9-c81acee6ebec-tls-certs\") pod \"832b0393-1d05-44ee-9da9-c81acee6ebec\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") "
Jul 29 10:43:55.292592 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.292401    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-home\") pod \"832b0393-1d05-44ee-9da9-c81acee6ebec\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") "
Jul 29 10:43:55.292592 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.292435    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-model-cache\") pod \"832b0393-1d05-44ee-9da9-c81acee6ebec\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") "
Jul 29 10:43:55.292592 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.292450    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-tmp-dir\") pod \"832b0393-1d05-44ee-9da9-c81acee6ebec\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") "
Jul 29 10:43:55.292592 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.292471    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-dshm\") pod \"832b0393-1d05-44ee-9da9-c81acee6ebec\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") "
Jul 29 10:43:55.292592 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.292513    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5tf5\" (UniqueName: \"kubernetes.io/projected/832b0393-1d05-44ee-9da9-c81acee6ebec-kube-api-access-h5tf5\") pod \"832b0393-1d05-44ee-9da9-c81acee6ebec\" (UID: \"832b0393-1d05-44ee-9da9-c81acee6ebec\") "
Jul 29 10:43:55.292870 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.292666    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-model-cache" (OuterVolumeSpecName: "model-cache") pod "832b0393-1d05-44ee-9da9-c81acee6ebec" (UID: "832b0393-1d05-44ee-9da9-c81acee6ebec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:43:55.292870 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.292702    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-home" (OuterVolumeSpecName: "home") pod "832b0393-1d05-44ee-9da9-c81acee6ebec" (UID: "832b0393-1d05-44ee-9da9-c81acee6ebec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:43:55.292870 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.292712    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "832b0393-1d05-44ee-9da9-c81acee6ebec" (UID: "832b0393-1d05-44ee-9da9-c81acee6ebec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:43:55.292870 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.292804    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-home\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:43:55.292870 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.292816    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-model-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:43:55.292870 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.292844    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:43:55.294658 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.294636    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-dshm" (OuterVolumeSpecName: "dshm") pod "832b0393-1d05-44ee-9da9-c81acee6ebec" (UID: "832b0393-1d05-44ee-9da9-c81acee6ebec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:43:55.294658 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.294648    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832b0393-1d05-44ee-9da9-c81acee6ebec-kube-api-access-h5tf5" (OuterVolumeSpecName: "kube-api-access-h5tf5") pod "832b0393-1d05-44ee-9da9-c81acee6ebec" (UID: "832b0393-1d05-44ee-9da9-c81acee6ebec"). InnerVolumeSpecName "kube-api-access-h5tf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:43:55.294838 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.294668    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/832b0393-1d05-44ee-9da9-c81acee6ebec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "832b0393-1d05-44ee-9da9-c81acee6ebec" (UID: "832b0393-1d05-44ee-9da9-c81acee6ebec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:43:55.393512 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.393480    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832b0393-1d05-44ee-9da9-c81acee6ebec-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:43:55.393512 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.393505    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832b0393-1d05-44ee-9da9-c81acee6ebec-dshm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:43:55.393512 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.393514    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5tf5\" (UniqueName: \"kubernetes.io/projected/832b0393-1d05-44ee-9da9-c81acee6ebec-kube-api-access-h5tf5\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:43:55.936351 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.936308    2584 generic.go:356] "Generic (PLEG): container finished" podID="832b0393-1d05-44ee-9da9-c81acee6ebec" containerID="e5955b38e586998b69bf134908f9d0dfb2aa5024117773b0e741d723a97b59f5" exitCode=0
Jul 29 10:43:55.936559 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.936360    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb" event={"ID":"832b0393-1d05-44ee-9da9-c81acee6ebec","Type":"ContainerDied","Data":"e5955b38e586998b69bf134908f9d0dfb2aa5024117773b0e741d723a97b59f5"}
Jul 29 10:43:55.936559 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.936412    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb" event={"ID":"832b0393-1d05-44ee-9da9-c81acee6ebec","Type":"ContainerDied","Data":"c84b857068a770364dc1148878084c929cd15922ae8e2caf1654e42060df3ce9"}
Jul 29 10:43:55.936559 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.936443    2584 scope.go:117] "RemoveContainer" containerID="e5955b38e586998b69bf134908f9d0dfb2aa5024117773b0e741d723a97b59f5"
Jul 29 10:43:55.936757 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.936610    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"
Jul 29 10:43:55.951891 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.951695    2584 scope.go:117] "RemoveContainer" containerID="e5955b38e586998b69bf134908f9d0dfb2aa5024117773b0e741d723a97b59f5"
Jul 29 10:43:55.953049 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:43:55.953011    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5955b38e586998b69bf134908f9d0dfb2aa5024117773b0e741d723a97b59f5\": container with ID starting with e5955b38e586998b69bf134908f9d0dfb2aa5024117773b0e741d723a97b59f5 not found: ID does not exist" containerID="e5955b38e586998b69bf134908f9d0dfb2aa5024117773b0e741d723a97b59f5"
Jul 29 10:43:55.953175 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.953055    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5955b38e586998b69bf134908f9d0dfb2aa5024117773b0e741d723a97b59f5"} err="failed to get container status \"e5955b38e586998b69bf134908f9d0dfb2aa5024117773b0e741d723a97b59f5\": rpc error: code = NotFound desc = could not find container \"e5955b38e586998b69bf134908f9d0dfb2aa5024117773b0e741d723a97b59f5\": container with ID starting with e5955b38e586998b69bf134908f9d0dfb2aa5024117773b0e741d723a97b59f5 not found: ID does not exist"
Jul 29 10:43:55.971115 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.971089    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"]
Jul 29 10:43:55.977486 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:55.977464    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-28ckb"]
Jul 29 10:43:56.257135 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:56.257058    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="832b0393-1d05-44ee-9da9-c81acee6ebec" path="/var/lib/kubelet/pods/832b0393-1d05-44ee-9da9-c81acee6ebec/volumes"
Jul 29 10:43:58.829788 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:58.829756    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"]
Jul 29 10:43:58.830169 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:58.829991    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95" podUID="a4551277-c244-420a-b4bf-c8ed35b3d9d5" containerName="istio-proxy" containerID="cri-o://c220f481c657fecf190b44ffc2eb33d6c0c16e1cd4f35295ad96d93e15eacfe1" gracePeriod=30
Jul 29 10:43:59.004781 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.004745    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"]
Jul 29 10:43:59.005090 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.005078    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="832b0393-1d05-44ee-9da9-c81acee6ebec" containerName="main"
Jul 29 10:43:59.005132 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.005092    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="832b0393-1d05-44ee-9da9-c81acee6ebec" containerName="main"
Jul 29 10:43:59.005166 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.005141    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="832b0393-1d05-44ee-9da9-c81acee6ebec" containerName="main"
Jul 29 10:43:59.009449 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.009430    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.012233 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.012208    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 29 10:43:59.012477 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.012452    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Jul 29 10:43:59.012606 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.012517    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 29 10:43:59.012806 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.012790    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-wfvw4\""
Jul 29 10:43:59.019451 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.019430    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"]
Jul 29 10:43:59.127392 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.127301    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rsct\" (UniqueName: \"kubernetes.io/projected/02de7692-b936-48f0-9cc9-84d5d823974f-kube-api-access-2rsct\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.127392 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.127351    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.127392 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.127368    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.127625 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.127469    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.127625 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.127498    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.127625 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.127548    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02de7692-b936-48f0-9cc9-84d5d823974f-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.228920 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.228891    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.228920 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.228925    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.229148 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.228970    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02de7692-b936-48f0-9cc9-84d5d823974f-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.229148 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.229030    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rsct\" (UniqueName: \"kubernetes.io/projected/02de7692-b936-48f0-9cc9-84d5d823974f-kube-api-access-2rsct\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.229148 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.229081    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.229148 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.229101    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.229376 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.229348    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.229451 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.229424    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.229513 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.229461    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.231214 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.231194    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.231478 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.231461    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02de7692-b936-48f0-9cc9-84d5d823974f-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.237091 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.237069    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rsct\" (UniqueName: \"kubernetes.io/projected/02de7692-b936-48f0-9cc9-84d5d823974f-kube-api-access-2rsct\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.321078 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.321042    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:43:59.459993 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.459966    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"]
Jul 29 10:43:59.462922 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:43:59.462897    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02de7692_b936_48f0_9cc9_84d5d823974f.slice/crio-c8ec65a875f7ce4fd8f32e2e0f91a95c11fbf9c2ccad0fc08867c34f0562e4f9 WatchSource:0}: Error finding container c8ec65a875f7ce4fd8f32e2e0f91a95c11fbf9c2ccad0fc08867c34f0562e4f9: Status 404 returned error can't find the container with id c8ec65a875f7ce4fd8f32e2e0f91a95c11fbf9c2ccad0fc08867c34f0562e4f9
Jul 29 10:43:59.960020 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.959988    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx" event={"ID":"02de7692-b936-48f0-9cc9-84d5d823974f","Type":"ContainerStarted","Data":"86782dbef81c197c6b128510241eb3206cb79119695c0ba53196fd082c173dde"}
Jul 29 10:43:59.960504 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.960038    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx" event={"ID":"02de7692-b936-48f0-9cc9-84d5d823974f","Type":"ContainerStarted","Data":"c8ec65a875f7ce4fd8f32e2e0f91a95c11fbf9c2ccad0fc08867c34f0562e4f9"}
Jul 29 10:43:59.980256 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:43:59.980192    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx" podStartSLOduration=1.980171612 podStartE2EDuration="1.980171612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:43:59.978573875 +0000 UTC m=+908.171601223" watchObservedRunningTime="2026-07-29 10:43:59.980171612 +0000 UTC m=+908.173198960"
Jul 29 10:44:00.321423 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.321378    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:44:00.720319 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.720283    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95" podUID="a4551277-c244-420a-b4bf-c8ed35b3d9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": read tcp 10.134.0.2:46984->10.134.0.26:15021: read: connection reset by peer"
Jul 29 10:44:00.874115 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.874092    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:44:00.945450 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.945421    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-credential-socket\") pod \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") "
Jul 29 10:44:00.945621 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.945457    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-podinfo\") pod \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") "
Jul 29 10:44:00.945621 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.945477    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-token\") pod \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") "
Jul 29 10:44:00.945621 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.945514    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-workload-socket\") pod \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") "
Jul 29 10:44:00.945621 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.945536    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-workload-certs\") pod \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") "
Jul 29 10:44:00.945621 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.945558    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istiod-ca-cert\") pod \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") "
Jul 29 10:44:00.945906 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.945624    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-envoy\") pod \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") "
Jul 29 10:44:00.945906 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.945668    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxrx4\" (UniqueName: \"kubernetes.io/projected/a4551277-c244-420a-b4bf-c8ed35b3d9d5-kube-api-access-nxrx4\") pod \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") "
Jul 29 10:44:00.945906 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.945734    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-data\") pod \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\" (UID: \"a4551277-c244-420a-b4bf-c8ed35b3d9d5\") "
Jul 29 10:44:00.945906 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.945784    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a4551277-c244-420a-b4bf-c8ed35b3d9d5" (UID: "a4551277-c244-420a-b4bf-c8ed35b3d9d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:00.945906 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.945811    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a4551277-c244-420a-b4bf-c8ed35b3d9d5" (UID: "a4551277-c244-420a-b4bf-c8ed35b3d9d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:00.946169 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.945957    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a4551277-c244-420a-b4bf-c8ed35b3d9d5" (UID: "a4551277-c244-420a-b4bf-c8ed35b3d9d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:00.946169 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.946031    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-credential-socket\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:44:00.946169 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.946051    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-workload-socket\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:44:00.946169 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.946067    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-workload-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:44:00.946169 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.946147    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-data" (OuterVolumeSpecName: "istio-data") pod "a4551277-c244-420a-b4bf-c8ed35b3d9d5" (UID: "a4551277-c244-420a-b4bf-c8ed35b3d9d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:00.946358 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.946079    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a4551277-c244-420a-b4bf-c8ed35b3d9d5" (UID: "a4551277-c244-420a-b4bf-c8ed35b3d9d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:44:00.947956 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.947911    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-token" (OuterVolumeSpecName: "istio-token") pod "a4551277-c244-420a-b4bf-c8ed35b3d9d5" (UID: "a4551277-c244-420a-b4bf-c8ed35b3d9d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:44:00.948089 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.948069    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a4551277-c244-420a-b4bf-c8ed35b3d9d5" (UID: "a4551277-c244-420a-b4bf-c8ed35b3d9d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 10:44:00.948212 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.948193    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a4551277-c244-420a-b4bf-c8ed35b3d9d5" (UID: "a4551277-c244-420a-b4bf-c8ed35b3d9d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:00.948326 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.948306    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4551277-c244-420a-b4bf-c8ed35b3d9d5-kube-api-access-nxrx4" (OuterVolumeSpecName: "kube-api-access-nxrx4") pod "a4551277-c244-420a-b4bf-c8ed35b3d9d5" (UID: "a4551277-c244-420a-b4bf-c8ed35b3d9d5"). InnerVolumeSpecName "kube-api-access-nxrx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:44:00.965769 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.965739    2584 generic.go:356] "Generic (PLEG): container finished" podID="a4551277-c244-420a-b4bf-c8ed35b3d9d5" containerID="c220f481c657fecf190b44ffc2eb33d6c0c16e1cd4f35295ad96d93e15eacfe1" exitCode=0
Jul 29 10:44:00.966084 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.965789    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"
Jul 29 10:44:00.966084 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.965822    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95" event={"ID":"a4551277-c244-420a-b4bf-c8ed35b3d9d5","Type":"ContainerDied","Data":"c220f481c657fecf190b44ffc2eb33d6c0c16e1cd4f35295ad96d93e15eacfe1"}
Jul 29 10:44:00.966084 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.965877    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95" event={"ID":"a4551277-c244-420a-b4bf-c8ed35b3d9d5","Type":"ContainerDied","Data":"aafbaf1ff5e7c34c5f83f5f26b1572cf31252e289577cee4db02e42bc115035d"}
Jul 29 10:44:00.966084 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.965901    2584 scope.go:117] "RemoveContainer" containerID="c220f481c657fecf190b44ffc2eb33d6c0c16e1cd4f35295ad96d93e15eacfe1"
Jul 29 10:44:00.974887 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.974870    2584 scope.go:117] "RemoveContainer" containerID="c220f481c657fecf190b44ffc2eb33d6c0c16e1cd4f35295ad96d93e15eacfe1"
Jul 29 10:44:00.975137 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:44:00.975114    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c220f481c657fecf190b44ffc2eb33d6c0c16e1cd4f35295ad96d93e15eacfe1\": container with ID starting with c220f481c657fecf190b44ffc2eb33d6c0c16e1cd4f35295ad96d93e15eacfe1 not found: ID does not exist" containerID="c220f481c657fecf190b44ffc2eb33d6c0c16e1cd4f35295ad96d93e15eacfe1"
Jul 29 10:44:00.975226 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.975141    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c220f481c657fecf190b44ffc2eb33d6c0c16e1cd4f35295ad96d93e15eacfe1"} err="failed to get container status \"c220f481c657fecf190b44ffc2eb33d6c0c16e1cd4f35295ad96d93e15eacfe1\": rpc error: code = NotFound desc = could not find container \"c220f481c657fecf190b44ffc2eb33d6c0c16e1cd4f35295ad96d93e15eacfe1\": container with ID starting with c220f481c657fecf190b44ffc2eb33d6c0c16e1cd4f35295ad96d93e15eacfe1 not found: ID does not exist"
Jul 29 10:44:00.988533 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.988493    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"]
Jul 29 10:44:00.990063 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:00.990045    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zqg95"]
Jul 29 10:44:01.047150 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:01.047119    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istiod-ca-cert\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:44:01.047150 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:01.047147    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-envoy\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:44:01.047150 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:01.047156    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxrx4\" (UniqueName: \"kubernetes.io/projected/a4551277-c244-420a-b4bf-c8ed35b3d9d5-kube-api-access-nxrx4\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:44:01.047452 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:01.047165    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-data\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:44:01.047452 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:01.047175    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-podinfo\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:44:01.047452 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:01.047184    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4551277-c244-420a-b4bf-c8ed35b3d9d5-istio-token\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:44:01.807858 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:01.807828    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"]
Jul 29 10:44:02.056618 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:44:02.056587    2584 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 29 10:44:02.057030 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:44:02.056661    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02de7692-b936-48f0-9cc9-84d5d823974f-tls-certs podName:02de7692-b936-48f0-9cc9-84d5d823974f nodeName:}" failed. No retries permitted until 2026-07-29 10:44:02.55664402 +0000 UTC m=+910.749671345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/02de7692-b936-48f0-9cc9-84d5d823974f-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx" (UID: "02de7692-b936-48f0-9cc9-84d5d823974f") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 29 10:44:02.254928 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:02.254832    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4551277-c244-420a-b4bf-c8ed35b3d9d5" path="/var/lib/kubelet/pods/a4551277-c244-420a-b4bf-c8ed35b3d9d5/volumes"
Jul 29 10:44:02.561769 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:44:02.561743    2584 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 29 10:44:02.561949 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:44:02.561809    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02de7692-b936-48f0-9cc9-84d5d823974f-tls-certs podName:02de7692-b936-48f0-9cc9-84d5d823974f nodeName:}" failed. No retries permitted until 2026-07-29 10:44:03.56179541 +0000 UTC m=+911.754822739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/02de7692-b936-48f0-9cc9-84d5d823974f-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx" (UID: "02de7692-b936-48f0-9cc9-84d5d823974f") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 29 10:44:02.983831 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:44:02.983748    2584 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:44:02.983831 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:44:02.983831 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:44:02.983831 ip-10-0-140-240 kubenswrapper[2584]:  > containerID="86782dbef81c197c6b128510241eb3206cb79119695c0ba53196fd082c173dde" cmd=["/bin/sleep","15"]
Jul 29 10:44:02.983831 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:02.983798    2584 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx" message="" err=<
Jul 29 10:44:02.983831 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:44:02.983831 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:44:02.983831 ip-10-0-140-240 kubenswrapper[2584]:  >
Jul 29 10:44:02.983831 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:44:02.983810    2584 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:44:02.983831 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:44:02.983831 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:44:02.983831 ip-10-0-140-240 kubenswrapper[2584]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx" podUID="02de7692-b936-48f0-9cc9-84d5d823974f" containerName="main" containerID="cri-o://86782dbef81c197c6b128510241eb3206cb79119695c0ba53196fd082c173dde"
Jul 29 10:44:02.984267 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:02.983854    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx" podUID="02de7692-b936-48f0-9cc9-84d5d823974f" containerName="main" containerID="cri-o://86782dbef81c197c6b128510241eb3206cb79119695c0ba53196fd082c173dde" gracePeriod=60
Jul 29 10:44:03.231172 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.231148    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:44:03.369529 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.369494    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02de7692-b936-48f0-9cc9-84d5d823974f-tls-certs\") pod \"02de7692-b936-48f0-9cc9-84d5d823974f\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") "
Jul 29 10:44:03.369732 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.369576    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-home\") pod \"02de7692-b936-48f0-9cc9-84d5d823974f\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") "
Jul 29 10:44:03.369732 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.369600    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-model-cache\") pod \"02de7692-b936-48f0-9cc9-84d5d823974f\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") "
Jul 29 10:44:03.369732 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.369642    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-dshm\") pod \"02de7692-b936-48f0-9cc9-84d5d823974f\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") "
Jul 29 10:44:03.369902 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.369743    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rsct\" (UniqueName: \"kubernetes.io/projected/02de7692-b936-48f0-9cc9-84d5d823974f-kube-api-access-2rsct\") pod \"02de7692-b936-48f0-9cc9-84d5d823974f\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") "
Jul 29 10:44:03.369902 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.369771    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-tmp-dir\") pod \"02de7692-b936-48f0-9cc9-84d5d823974f\" (UID: \"02de7692-b936-48f0-9cc9-84d5d823974f\") "
Jul 29 10:44:03.370014 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.369935    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-model-cache" (OuterVolumeSpecName: "model-cache") pod "02de7692-b936-48f0-9cc9-84d5d823974f" (UID: "02de7692-b936-48f0-9cc9-84d5d823974f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:03.370014 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.369945    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-home" (OuterVolumeSpecName: "home") pod "02de7692-b936-48f0-9cc9-84d5d823974f" (UID: "02de7692-b936-48f0-9cc9-84d5d823974f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:03.370120 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.370055    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-home\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:44:03.370120 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.370073    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-model-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:44:03.370213 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.370139    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "02de7692-b936-48f0-9cc9-84d5d823974f" (UID: "02de7692-b936-48f0-9cc9-84d5d823974f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:03.371936 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.371908    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02de7692-b936-48f0-9cc9-84d5d823974f-kube-api-access-2rsct" (OuterVolumeSpecName: "kube-api-access-2rsct") pod "02de7692-b936-48f0-9cc9-84d5d823974f" (UID: "02de7692-b936-48f0-9cc9-84d5d823974f"). InnerVolumeSpecName "kube-api-access-2rsct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:44:03.371936 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.371914    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02de7692-b936-48f0-9cc9-84d5d823974f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02de7692-b936-48f0-9cc9-84d5d823974f" (UID: "02de7692-b936-48f0-9cc9-84d5d823974f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:44:03.372117 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.372098    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-dshm" (OuterVolumeSpecName: "dshm") pod "02de7692-b936-48f0-9cc9-84d5d823974f" (UID: "02de7692-b936-48f0-9cc9-84d5d823974f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:03.470531 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.470490    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rsct\" (UniqueName: \"kubernetes.io/projected/02de7692-b936-48f0-9cc9-84d5d823974f-kube-api-access-2rsct\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:44:03.470531 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.470521    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:44:03.470531 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.470540    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02de7692-b936-48f0-9cc9-84d5d823974f-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:44:03.470789 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.470551    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02de7692-b936-48f0-9cc9-84d5d823974f-dshm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:44:03.980558 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.980530    2584 generic.go:356] "Generic (PLEG): container finished" podID="02de7692-b936-48f0-9cc9-84d5d823974f" containerID="86782dbef81c197c6b128510241eb3206cb79119695c0ba53196fd082c173dde" exitCode=0
Jul 29 10:44:03.980765 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.980569    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"
Jul 29 10:44:03.980765 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.980618    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx" event={"ID":"02de7692-b936-48f0-9cc9-84d5d823974f","Type":"ContainerDied","Data":"86782dbef81c197c6b128510241eb3206cb79119695c0ba53196fd082c173dde"}
Jul 29 10:44:03.980765 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.980696    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx" event={"ID":"02de7692-b936-48f0-9cc9-84d5d823974f","Type":"ContainerDied","Data":"c8ec65a875f7ce4fd8f32e2e0f91a95c11fbf9c2ccad0fc08867c34f0562e4f9"}
Jul 29 10:44:03.980765 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.980717    2584 scope.go:117] "RemoveContainer" containerID="86782dbef81c197c6b128510241eb3206cb79119695c0ba53196fd082c173dde"
Jul 29 10:44:03.989988 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.989837    2584 scope.go:117] "RemoveContainer" containerID="86782dbef81c197c6b128510241eb3206cb79119695c0ba53196fd082c173dde"
Jul 29 10:44:03.990142 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:44:03.990120    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86782dbef81c197c6b128510241eb3206cb79119695c0ba53196fd082c173dde\": container with ID starting with 86782dbef81c197c6b128510241eb3206cb79119695c0ba53196fd082c173dde not found: ID does not exist" containerID="86782dbef81c197c6b128510241eb3206cb79119695c0ba53196fd082c173dde"
Jul 29 10:44:03.990184 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:03.990151    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86782dbef81c197c6b128510241eb3206cb79119695c0ba53196fd082c173dde"} err="failed to get container status \"86782dbef81c197c6b128510241eb3206cb79119695c0ba53196fd082c173dde\": rpc error: code = NotFound desc = could not find container \"86782dbef81c197c6b128510241eb3206cb79119695c0ba53196fd082c173dde\": container with ID starting with 86782dbef81c197c6b128510241eb3206cb79119695c0ba53196fd082c173dde not found: ID does not exist"
Jul 29 10:44:04.002500 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:04.002473    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"]
Jul 29 10:44:04.005875 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:04.005852    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffc8zx"]
Jul 29 10:44:04.255170 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:04.255091    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02de7692-b936-48f0-9cc9-84d5d823974f" path="/var/lib/kubelet/pods/02de7692-b936-48f0-9cc9-84d5d823974f/volumes"
Jul 29 10:44:08.020341 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.020311    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"]
Jul 29 10:44:08.020731 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.020633    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02de7692-b936-48f0-9cc9-84d5d823974f" containerName="main"
Jul 29 10:44:08.020731 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.020645    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="02de7692-b936-48f0-9cc9-84d5d823974f" containerName="main"
Jul 29 10:44:08.020731 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.020696    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4551277-c244-420a-b4bf-c8ed35b3d9d5" containerName="istio-proxy"
Jul 29 10:44:08.020731 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.020704    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4551277-c244-420a-b4bf-c8ed35b3d9d5" containerName="istio-proxy"
Jul 29 10:44:08.020860 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.020754    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4551277-c244-420a-b4bf-c8ed35b3d9d5" containerName="istio-proxy"
Jul 29 10:44:08.020860 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.020763    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="02de7692-b936-48f0-9cc9-84d5d823974f" containerName="main"
Jul 29 10:44:08.024999 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.024984    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.028063 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.028038    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Jul 29 10:44:08.028201 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.028180    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 29 10:44:08.028242 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.028180    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-ncs6p\""
Jul 29 10:44:08.028315 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.028297    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Jul 29 10:44:08.029373 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.029355    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Jul 29 10:44:08.035048 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.035025    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"]
Jul 29 10:44:08.115248 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.115209    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnzvl\" (UniqueName: \"kubernetes.io/projected/df95f6ae-3f82-44a5-80f4-246668a970be-kube-api-access-vnzvl\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.115417 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.115261    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-home\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.115417 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.115316    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-tmp-dir\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.115417 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.115364    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-kserve-provision-location\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.115417 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.115406    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-model-cache\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.115588 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.115460    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df95f6ae-3f82-44a5-80f4-246668a970be-tls-certs\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.115588 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.115510    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-dshm\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.216805 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.216768    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-home\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.216805 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.216812    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-tmp-dir\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.217079 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.216835    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-kserve-provision-location\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.217079 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.216874    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-model-cache\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.217079 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.216893    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df95f6ae-3f82-44a5-80f4-246668a970be-tls-certs\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.217079 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.216934    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-dshm\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.217079 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.216986    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnzvl\" (UniqueName: \"kubernetes.io/projected/df95f6ae-3f82-44a5-80f4-246668a970be-kube-api-access-vnzvl\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.217321 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.217134    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-home\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.217460 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.217437    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-kserve-provision-location\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.217527 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.217457    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-tmp-dir\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.217527 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.217511    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-model-cache\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.219465 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.219441    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-dshm\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.219592 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.219539    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df95f6ae-3f82-44a5-80f4-246668a970be-tls-certs\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.230636 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.230613    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnzvl\" (UniqueName: \"kubernetes.io/projected/df95f6ae-3f82-44a5-80f4-246668a970be-kube-api-access-vnzvl\") pod \"auth-enabled-test-kserve-7f985955d4-rwmtj\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.335379 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.335347    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:08.467581 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.467551    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"]
Jul 29 10:44:08.470881 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:44:08.470854    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf95f6ae_3f82_44a5_80f4_246668a970be.slice/crio-6494b83af53c2f20c65e95e2257366a3564921fe69a3d80d41b26839f7919762 WatchSource:0}: Error finding container 6494b83af53c2f20c65e95e2257366a3564921fe69a3d80d41b26839f7919762: Status 404 returned error can't find the container with id 6494b83af53c2f20c65e95e2257366a3564921fe69a3d80d41b26839f7919762
Jul 29 10:44:08.472836 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:08.472821    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:44:09.005841 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:09.005804    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" event={"ID":"df95f6ae-3f82-44a5-80f4-246668a970be","Type":"ContainerStarted","Data":"6494b83af53c2f20c65e95e2257366a3564921fe69a3d80d41b26839f7919762"}
Jul 29 10:44:13.028056 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:13.028016    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" event={"ID":"df95f6ae-3f82-44a5-80f4-246668a970be","Type":"ContainerStarted","Data":"d080974bd9686097d09f8c46d89029aebf84fe9ada00a3e48d4a65ceaa8af0dd"}
Jul 29 10:44:17.047791 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:17.047760    2584 generic.go:356] "Generic (PLEG): container finished" podID="df95f6ae-3f82-44a5-80f4-246668a970be" containerID="d080974bd9686097d09f8c46d89029aebf84fe9ada00a3e48d4a65ceaa8af0dd" exitCode=0
Jul 29 10:44:17.048200 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:17.047807    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" event={"ID":"df95f6ae-3f82-44a5-80f4-246668a970be","Type":"ContainerDied","Data":"d080974bd9686097d09f8c46d89029aebf84fe9ada00a3e48d4a65ceaa8af0dd"}
Jul 29 10:44:46.214633 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:46.214602    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" event={"ID":"df95f6ae-3f82-44a5-80f4-246668a970be","Type":"ContainerStarted","Data":"4c739e3e99633fa717333393bb5d5e97e5f7585dd98351acd6f335d89461b5ec"}
Jul 29 10:44:46.238461 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:46.238391    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podStartSLOduration=1.854971398 podStartE2EDuration="38.238371653s" totalImagesPullingTime="32.33121458s" totalInitContainerRuntime="4.052185675s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:44:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:44:46.236387689 +0000 UTC m=+954.429415036" watchObservedRunningTime="2026-07-29 10:44:46.238371653 +0000 UTC m=+954.431399000"
Jul 29 10:44:46.338809 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:46.338761    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:48.335483 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:48.335441    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:44:48.337070 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:48.337037    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 10:44:58.336044 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:44:58.336001    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 10:45:08.336568 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:08.336523    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 10:45:11.712259 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.712222    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 29 10:45:11.759137 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.759104    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 29 10:45:11.759309 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.759252    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.762820 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.762795    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 29 10:45:11.762994 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.762891    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-hk947\""
Jul 29 10:45:11.763224 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.763209    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-5v7b2\""
Jul 29 10:45:11.763911 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.763888    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 29 10:45:11.764015 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.763888    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 29 10:45:11.764015 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.763989    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 29 10:45:11.807582 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.807550    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.807781 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.807606    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.807781 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.807658    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.807781 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.807708    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.807781 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.807737    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.807781 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.807771    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.808001 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.807835    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvvm\" (UniqueName: \"kubernetes.io/projected/cda94357-9166-4903-bda2-fc37b595478a-kube-api-access-mmvvm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.909126 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.909092    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.909126 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.909141    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.909392 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.909167    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.909392 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.909203    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.909392 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.909235    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.909392 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.909273    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvvm\" (UniqueName: \"kubernetes.io/projected/cda94357-9166-4903-bda2-fc37b595478a-kube-api-access-mmvvm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.909392 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.909323    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.909663 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.909514    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.909663 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.909558    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.909663 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.909589    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.909838 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.909785    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.911629 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.911597    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.911797 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.911746    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:11.917671 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:11.917652    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvvm\" (UniqueName: \"kubernetes.io/projected/cda94357-9166-4903-bda2-fc37b595478a-kube-api-access-mmvvm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:12.070188 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:12.070153    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:45:12.218825 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:12.218790    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 29 10:45:12.220981 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:45:12.220946    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda94357_9166_4903_bda2_fc37b595478a.slice/crio-5437c38c486fc44afadd11400ab04a259ad3d2a5d2e69143d671bc45164a57c2 WatchSource:0}: Error finding container 5437c38c486fc44afadd11400ab04a259ad3d2a5d2e69143d671bc45164a57c2: Status 404 returned error can't find the container with id 5437c38c486fc44afadd11400ab04a259ad3d2a5d2e69143d671bc45164a57c2
Jul 29 10:45:12.334069 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:12.333977    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"cda94357-9166-4903-bda2-fc37b595478a","Type":"ContainerStarted","Data":"3de9c9b999d7ad5ad271f901a972136d8326fee9888fbb505c50ed89646745f4"}
Jul 29 10:45:12.334069 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:12.334041    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"cda94357-9166-4903-bda2-fc37b595478a","Type":"ContainerStarted","Data":"5437c38c486fc44afadd11400ab04a259ad3d2a5d2e69143d671bc45164a57c2"}
Jul 29 10:45:16.358327 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:16.358294    2584 generic.go:356] "Generic (PLEG): container finished" podID="cda94357-9166-4903-bda2-fc37b595478a" containerID="3de9c9b999d7ad5ad271f901a972136d8326fee9888fbb505c50ed89646745f4" exitCode=0
Jul 29 10:45:16.358772 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:16.358382    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"cda94357-9166-4903-bda2-fc37b595478a","Type":"ContainerDied","Data":"3de9c9b999d7ad5ad271f901a972136d8326fee9888fbb505c50ed89646745f4"}
Jul 29 10:45:17.366028 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:17.365985    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"cda94357-9166-4903-bda2-fc37b595478a","Type":"ContainerStarted","Data":"496602db655dda1ed0a1e48ac3de2894a4adf98db942a987d300b88d621cb12f"}
Jul 29 10:45:17.389082 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:17.389028    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=3.041495897 podStartE2EDuration="6.389013528s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.347517631s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:45:17.386614963 +0000 UTC m=+985.579642308" watchObservedRunningTime="2026-07-29 10:45:17.389013528 +0000 UTC m=+985.582040874"
Jul 29 10:45:18.336619 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:18.336573    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 10:45:28.336787 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:28.336735    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 10:45:38.336014 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:38.335968    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 10:45:48.336488 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:48.336442    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 10:45:58.336077 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:45:58.336037    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 10:46:08.336439 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:46:08.336384    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 10:46:18.336228 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:46:18.336184    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 10:46:28.336081 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:46:28.336036    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 10:46:38.336434 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:46:38.336383    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 10:46:48.336507 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:46:48.336465    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 10:46:58.336153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:46:58.336114    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 10:47:08.336702 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:08.336654    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 29 10:47:18.346510 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:18.346479    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:47:18.354455 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:18.354430    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:47:24.298956 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:24.298922    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"]
Jul 29 10:47:36.250612 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:36.250576    2584 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-hk947\" not found"
Jul 29 10:47:36.304370 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:47:36.304338    2584 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 10:47:36.304581 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:47:36.304401    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs podName:cda94357-9166-4903-bda2-fc37b595478a nodeName:}" failed. No retries permitted until 2026-07-29 10:47:36.804386634 +0000 UTC m=+1124.997413958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "cda94357-9166-4903-bda2-fc37b595478a") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 10:47:36.808998 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:47:36.808966    2584 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 10:47:36.809180 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:47:36.809067    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs podName:cda94357-9166-4903-bda2-fc37b595478a nodeName:}" failed. No retries permitted until 2026-07-29 10:47:37.809052356 +0000 UTC m=+1126.002079681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "cda94357-9166-4903-bda2-fc37b595478a") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 10:47:37.818735 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:47:37.818702    2584 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 10:47:37.819129 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:47:37.818785    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs podName:cda94357-9166-4903-bda2-fc37b595478a nodeName:}" failed. No retries permitted until 2026-07-29 10:47:39.8187663 +0000 UTC m=+1128.011793644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "cda94357-9166-4903-bda2-fc37b595478a") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 10:47:39.323440 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:39.323404    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main" containerID="cri-o://4c739e3e99633fa717333393bb5d5e97e5f7585dd98351acd6f335d89461b5ec" gracePeriod=45
Jul 29 10:47:39.836693 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:47:39.836638    2584 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 10:47:39.836843 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:47:39.836728    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs podName:cda94357-9166-4903-bda2-fc37b595478a nodeName:}" failed. No retries permitted until 2026-07-29 10:47:43.836708538 +0000 UTC m=+1132.029735867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "cda94357-9166-4903-bda2-fc37b595478a") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 10:47:40.078065 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.078042    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:47:40.094389 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.094319    2584 generic.go:356] "Generic (PLEG): container finished" podID="df95f6ae-3f82-44a5-80f4-246668a970be" containerID="4c739e3e99633fa717333393bb5d5e97e5f7585dd98351acd6f335d89461b5ec" exitCode=0
Jul 29 10:47:40.094389 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.094376    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"
Jul 29 10:47:40.094590 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.094403    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" event={"ID":"df95f6ae-3f82-44a5-80f4-246668a970be","Type":"ContainerDied","Data":"4c739e3e99633fa717333393bb5d5e97e5f7585dd98351acd6f335d89461b5ec"}
Jul 29 10:47:40.094590 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.094460    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj" event={"ID":"df95f6ae-3f82-44a5-80f4-246668a970be","Type":"ContainerDied","Data":"6494b83af53c2f20c65e95e2257366a3564921fe69a3d80d41b26839f7919762"}
Jul 29 10:47:40.094590 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.094487    2584 scope.go:117] "RemoveContainer" containerID="4c739e3e99633fa717333393bb5d5e97e5f7585dd98351acd6f335d89461b5ec"
Jul 29 10:47:40.103428 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.103406    2584 scope.go:117] "RemoveContainer" containerID="d080974bd9686097d09f8c46d89029aebf84fe9ada00a3e48d4a65ceaa8af0dd"
Jul 29 10:47:40.112070 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.112053    2584 scope.go:117] "RemoveContainer" containerID="4c739e3e99633fa717333393bb5d5e97e5f7585dd98351acd6f335d89461b5ec"
Jul 29 10:47:40.112329 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:47:40.112312    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c739e3e99633fa717333393bb5d5e97e5f7585dd98351acd6f335d89461b5ec\": container with ID starting with 4c739e3e99633fa717333393bb5d5e97e5f7585dd98351acd6f335d89461b5ec not found: ID does not exist" containerID="4c739e3e99633fa717333393bb5d5e97e5f7585dd98351acd6f335d89461b5ec"
Jul 29 10:47:40.112379 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.112342    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c739e3e99633fa717333393bb5d5e97e5f7585dd98351acd6f335d89461b5ec"} err="failed to get container status \"4c739e3e99633fa717333393bb5d5e97e5f7585dd98351acd6f335d89461b5ec\": rpc error: code = NotFound desc = could not find container \"4c739e3e99633fa717333393bb5d5e97e5f7585dd98351acd6f335d89461b5ec\": container with ID starting with 4c739e3e99633fa717333393bb5d5e97e5f7585dd98351acd6f335d89461b5ec not found: ID does not exist"
Jul 29 10:47:40.112379 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.112359    2584 scope.go:117] "RemoveContainer" containerID="d080974bd9686097d09f8c46d89029aebf84fe9ada00a3e48d4a65ceaa8af0dd"
Jul 29 10:47:40.112582 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:47:40.112565    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d080974bd9686097d09f8c46d89029aebf84fe9ada00a3e48d4a65ceaa8af0dd\": container with ID starting with d080974bd9686097d09f8c46d89029aebf84fe9ada00a3e48d4a65ceaa8af0dd not found: ID does not exist" containerID="d080974bd9686097d09f8c46d89029aebf84fe9ada00a3e48d4a65ceaa8af0dd"
Jul 29 10:47:40.112638 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.112591    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d080974bd9686097d09f8c46d89029aebf84fe9ada00a3e48d4a65ceaa8af0dd"} err="failed to get container status \"d080974bd9686097d09f8c46d89029aebf84fe9ada00a3e48d4a65ceaa8af0dd\": rpc error: code = NotFound desc = could not find container \"d080974bd9686097d09f8c46d89029aebf84fe9ada00a3e48d4a65ceaa8af0dd\": container with ID starting with d080974bd9686097d09f8c46d89029aebf84fe9ada00a3e48d4a65ceaa8af0dd not found: ID does not exist"
Jul 29 10:47:40.139066 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.139038    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-tmp-dir\") pod \"df95f6ae-3f82-44a5-80f4-246668a970be\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") "
Jul 29 10:47:40.139182 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.139072    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-dshm\") pod \"df95f6ae-3f82-44a5-80f4-246668a970be\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") "
Jul 29 10:47:40.139182 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.139102    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnzvl\" (UniqueName: \"kubernetes.io/projected/df95f6ae-3f82-44a5-80f4-246668a970be-kube-api-access-vnzvl\") pod \"df95f6ae-3f82-44a5-80f4-246668a970be\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") "
Jul 29 10:47:40.139182 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.139125    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df95f6ae-3f82-44a5-80f4-246668a970be-tls-certs\") pod \"df95f6ae-3f82-44a5-80f4-246668a970be\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") "
Jul 29 10:47:40.139182 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.139179    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-home\") pod \"df95f6ae-3f82-44a5-80f4-246668a970be\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") "
Jul 29 10:47:40.139376 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.139246    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-kserve-provision-location\") pod \"df95f6ae-3f82-44a5-80f4-246668a970be\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") "
Jul 29 10:47:40.139376 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.139274    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-model-cache\") pod \"df95f6ae-3f82-44a5-80f4-246668a970be\" (UID: \"df95f6ae-3f82-44a5-80f4-246668a970be\") "
Jul 29 10:47:40.139825 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.139774    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-model-cache" (OuterVolumeSpecName: "model-cache") pod "df95f6ae-3f82-44a5-80f4-246668a970be" (UID: "df95f6ae-3f82-44a5-80f4-246668a970be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:47:40.140058 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.140027    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-home" (OuterVolumeSpecName: "home") pod "df95f6ae-3f82-44a5-80f4-246668a970be" (UID: "df95f6ae-3f82-44a5-80f4-246668a970be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:47:40.141395 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.141362    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df95f6ae-3f82-44a5-80f4-246668a970be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df95f6ae-3f82-44a5-80f4-246668a970be" (UID: "df95f6ae-3f82-44a5-80f4-246668a970be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:47:40.141395 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.141371    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-dshm" (OuterVolumeSpecName: "dshm") pod "df95f6ae-3f82-44a5-80f4-246668a970be" (UID: "df95f6ae-3f82-44a5-80f4-246668a970be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:47:40.141556 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.141424    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df95f6ae-3f82-44a5-80f4-246668a970be-kube-api-access-vnzvl" (OuterVolumeSpecName: "kube-api-access-vnzvl") pod "df95f6ae-3f82-44a5-80f4-246668a970be" (UID: "df95f6ae-3f82-44a5-80f4-246668a970be"). InnerVolumeSpecName "kube-api-access-vnzvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:47:40.152211 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.152185    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df95f6ae-3f82-44a5-80f4-246668a970be" (UID: "df95f6ae-3f82-44a5-80f4-246668a970be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:47:40.195063 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.195027    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df95f6ae-3f82-44a5-80f4-246668a970be" (UID: "df95f6ae-3f82-44a5-80f4-246668a970be"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:47:40.240319 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.240291    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:47:40.240319 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.240316    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-dshm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:47:40.240460 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.240326    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnzvl\" (UniqueName: \"kubernetes.io/projected/df95f6ae-3f82-44a5-80f4-246668a970be-kube-api-access-vnzvl\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:47:40.240460 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.240333    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df95f6ae-3f82-44a5-80f4-246668a970be-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:47:40.240460 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.240341    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-home\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:47:40.240460 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.240349    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-kserve-provision-location\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:47:40.240460 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.240357    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df95f6ae-3f82-44a5-80f4-246668a970be-model-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:47:40.412141 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.412045    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"]
Jul 29 10:47:40.416549 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:40.416525    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7f985955d4-rwmtj"]
Jul 29 10:47:42.255378 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:42.255344    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" path="/var/lib/kubelet/pods/df95f6ae-3f82-44a5-80f4-246668a970be/volumes"
Jul 29 10:47:43.873914 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:47:43.873876    2584 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 10:47:43.874306 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:47:43.873961    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs podName:cda94357-9166-4903-bda2-fc37b595478a nodeName:}" failed. No retries permitted until 2026-07-29 10:47:51.873944003 +0000 UTC m=+1140.066971343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "cda94357-9166-4903-bda2-fc37b595478a") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 10:47:47.430019 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:47:47.429971    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 29 10:47:51.945250 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:47:51.945214    2584 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 10:47:51.945652 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:47:51.945289    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs podName:cda94357-9166-4903-bda2-fc37b595478a nodeName:}" failed. No retries permitted until 2026-07-29 10:48:07.945274529 +0000 UTC m=+1156.138301853 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "cda94357-9166-4903-bda2-fc37b595478a") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 29 10:48:02.455486 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:02.455426    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="cda94357-9166-4903-bda2-fc37b595478a" containerName="main" containerID="cri-o://496602db655dda1ed0a1e48ac3de2894a4adf98db942a987d300b88d621cb12f" gracePeriod=45
Jul 29 10:48:03.198844 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.198815    2584 generic.go:356] "Generic (PLEG): container finished" podID="cda94357-9166-4903-bda2-fc37b595478a" containerID="496602db655dda1ed0a1e48ac3de2894a4adf98db942a987d300b88d621cb12f" exitCode=0
Jul 29 10:48:03.198980 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.198886    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"cda94357-9166-4903-bda2-fc37b595478a","Type":"ContainerDied","Data":"496602db655dda1ed0a1e48ac3de2894a4adf98db942a987d300b88d621cb12f"}
Jul 29 10:48:03.198980 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.198945    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"cda94357-9166-4903-bda2-fc37b595478a","Type":"ContainerDied","Data":"5437c38c486fc44afadd11400ab04a259ad3d2a5d2e69143d671bc45164a57c2"}
Jul 29 10:48:03.198980 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.198962    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5437c38c486fc44afadd11400ab04a259ad3d2a5d2e69143d671bc45164a57c2"
Jul 29 10:48:03.207652 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.207631    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:48:03.350798 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.350763    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-home\") pod \"cda94357-9166-4903-bda2-fc37b595478a\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") "
Jul 29 10:48:03.350798 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.350801    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-kserve-provision-location\") pod \"cda94357-9166-4903-bda2-fc37b595478a\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") "
Jul 29 10:48:03.351014 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.350832    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-dshm\") pod \"cda94357-9166-4903-bda2-fc37b595478a\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") "
Jul 29 10:48:03.351014 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.350896    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs\") pod \"cda94357-9166-4903-bda2-fc37b595478a\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") "
Jul 29 10:48:03.351014 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.350913    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-tmp-dir\") pod \"cda94357-9166-4903-bda2-fc37b595478a\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") "
Jul 29 10:48:03.351014 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.350927    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-model-cache\") pod \"cda94357-9166-4903-bda2-fc37b595478a\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") "
Jul 29 10:48:03.351014 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.350950    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmvvm\" (UniqueName: \"kubernetes.io/projected/cda94357-9166-4903-bda2-fc37b595478a-kube-api-access-mmvvm\") pod \"cda94357-9166-4903-bda2-fc37b595478a\" (UID: \"cda94357-9166-4903-bda2-fc37b595478a\") "
Jul 29 10:48:03.351406 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.351373    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-model-cache" (OuterVolumeSpecName: "model-cache") pod "cda94357-9166-4903-bda2-fc37b595478a" (UID: "cda94357-9166-4903-bda2-fc37b595478a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:48:03.351576 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.351551    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-home" (OuterVolumeSpecName: "home") pod "cda94357-9166-4903-bda2-fc37b595478a" (UID: "cda94357-9166-4903-bda2-fc37b595478a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:48:03.353261 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.353230    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cda94357-9166-4903-bda2-fc37b595478a" (UID: "cda94357-9166-4903-bda2-fc37b595478a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:48:03.353375 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.353291    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda94357-9166-4903-bda2-fc37b595478a-kube-api-access-mmvvm" (OuterVolumeSpecName: "kube-api-access-mmvvm") pod "cda94357-9166-4903-bda2-fc37b595478a" (UID: "cda94357-9166-4903-bda2-fc37b595478a"). InnerVolumeSpecName "kube-api-access-mmvvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:48:03.353375 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.353347    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-dshm" (OuterVolumeSpecName: "dshm") pod "cda94357-9166-4903-bda2-fc37b595478a" (UID: "cda94357-9166-4903-bda2-fc37b595478a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:48:03.371362 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.371325    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cda94357-9166-4903-bda2-fc37b595478a" (UID: "cda94357-9166-4903-bda2-fc37b595478a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:48:03.408719 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.408643    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cda94357-9166-4903-bda2-fc37b595478a" (UID: "cda94357-9166-4903-bda2-fc37b595478a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:48:03.451866 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.451834    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cda94357-9166-4903-bda2-fc37b595478a-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:48:03.451866 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.451860    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:48:03.451866 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.451869    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-model-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:48:03.452073 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.451882    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmvvm\" (UniqueName: \"kubernetes.io/projected/cda94357-9166-4903-bda2-fc37b595478a-kube-api-access-mmvvm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:48:03.452073 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.451899    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-home\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:48:03.452073 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.451907    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-kserve-provision-location\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:48:03.452073 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:03.451917    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cda94357-9166-4903-bda2-fc37b595478a-dshm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:48:04.203319 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:04.203288    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 29 10:48:04.228937 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:04.228900    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 29 10:48:04.232818 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:04.232793    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 29 10:48:04.254766 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:04.254727    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cda94357-9166-4903-bda2-fc37b595478a" path="/var/lib/kubelet/pods/cda94357-9166-4903-bda2-fc37b595478a/volumes"
Jul 29 10:48:16.965279 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.965189    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"]
Jul 29 10:48:16.965884 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.965667    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="storage-initializer"
Jul 29 10:48:16.965884 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.965705    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="storage-initializer"
Jul 29 10:48:16.965884 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.965718    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cda94357-9166-4903-bda2-fc37b595478a" containerName="storage-initializer"
Jul 29 10:48:16.965884 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.965726    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cda94357-9166-4903-bda2-fc37b595478a" containerName="storage-initializer"
Jul 29 10:48:16.965884 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.965753    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main"
Jul 29 10:48:16.965884 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.965763    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main"
Jul 29 10:48:16.965884 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.965782    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cda94357-9166-4903-bda2-fc37b595478a" containerName="main"
Jul 29 10:48:16.965884 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.965790    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cda94357-9166-4903-bda2-fc37b595478a" containerName="main"
Jul 29 10:48:16.965884 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.965882    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cda94357-9166-4903-bda2-fc37b595478a" containerName="main"
Jul 29 10:48:16.966316 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.965899    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="df95f6ae-3f82-44a5-80f4-246668a970be" containerName="main"
Jul 29 10:48:16.970532 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.970505    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:16.973550 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.973512    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Jul 29 10:48:16.973694 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.973570    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 29 10:48:16.973694 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.973585    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 29 10:48:16.973694 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.973628    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-b44q8\""
Jul 29 10:48:16.973694 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.973662    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 29 10:48:16.982433 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:16.982406    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"]
Jul 29 10:48:17.062163 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.062126    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-tls-certs\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.062344 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.062169    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bcsp\" (UniqueName: \"kubernetes.io/projected/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-kube-api-access-2bcsp\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.062344 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.062263    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-model-cache\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.062344 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.062291    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-tmp-dir\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.062524 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.062341    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-home\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.062524 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.062370    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.062524 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.062441    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-dshm\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.163133 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.163093    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-tls-certs\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.163133 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.163135    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bcsp\" (UniqueName: \"kubernetes.io/projected/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-kube-api-access-2bcsp\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.163380 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.163185    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-model-cache\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.163380 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.163202    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-tmp-dir\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.163380 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.163225    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-home\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.163380 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.163248    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.163380 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.163289    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-dshm\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.163688 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.163655    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-model-cache\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.163829 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.163743    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.163829 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.163804    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-tmp-dir\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.163829 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.163814    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-home\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.165563 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.165536    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-dshm\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.165950 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.165930    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-tls-certs\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.173131 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.173109    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bcsp\" (UniqueName: \"kubernetes.io/projected/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-kube-api-access-2bcsp\") pod \"auth-invalid-token-test-kserve-6977dd4757-nxfxx\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.283476 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.283399    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:17.425057 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:17.425023    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"]
Jul 29 10:48:17.428395 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:48:17.428367    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7778f8b5_6b3b_484c_8d88_0fac8e8242d3.slice/crio-93097be976c5df8de2b4294aff3733404ca9d6de59d81555b8df924932f7d550 WatchSource:0}: Error finding container 93097be976c5df8de2b4294aff3733404ca9d6de59d81555b8df924932f7d550: Status 404 returned error can't find the container with id 93097be976c5df8de2b4294aff3733404ca9d6de59d81555b8df924932f7d550
Jul 29 10:48:18.267646 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:18.267613    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" event={"ID":"7778f8b5-6b3b-484c-8d88-0fac8e8242d3","Type":"ContainerStarted","Data":"43eb6f3b6965b658184e3dfae9ede25eaa230ac55dd0c0193af435163bfeb5a6"}
Jul 29 10:48:18.268052 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:18.267668    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" event={"ID":"7778f8b5-6b3b-484c-8d88-0fac8e8242d3","Type":"ContainerStarted","Data":"93097be976c5df8de2b4294aff3733404ca9d6de59d81555b8df924932f7d550"}
Jul 29 10:48:22.288700 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:22.288647    2584 generic.go:356] "Generic (PLEG): container finished" podID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerID="43eb6f3b6965b658184e3dfae9ede25eaa230ac55dd0c0193af435163bfeb5a6" exitCode=0
Jul 29 10:48:22.289289 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:22.288728    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" event={"ID":"7778f8b5-6b3b-484c-8d88-0fac8e8242d3","Type":"ContainerDied","Data":"43eb6f3b6965b658184e3dfae9ede25eaa230ac55dd0c0193af435163bfeb5a6"}
Jul 29 10:48:23.295067 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:23.295034    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" event={"ID":"7778f8b5-6b3b-484c-8d88-0fac8e8242d3","Type":"ContainerStarted","Data":"99fc739175c4f2e22491a706fe9bbafde77b009139600288f624bbd6c0d8abfe"}
Jul 29 10:48:23.333347 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:23.333277    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" podStartSLOduration=2.652790435 podStartE2EDuration="7.333254043s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.680463608s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:48:23.331124119 +0000 UTC m=+1171.524151467" watchObservedRunningTime="2026-07-29 10:48:23.333254043 +0000 UTC m=+1171.526281391"
Jul 29 10:48:24.283934 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:24.283898    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:27.284002 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:27.283961    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:48:27.285604 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:27.285574    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 29 10:48:37.284489 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:37.284436    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 29 10:48:47.284203 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:47.284153    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 29 10:48:57.284279 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:48:57.284231    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 29 10:49:07.284426 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:49:07.284382    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 29 10:49:17.284564 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:49:17.284516    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 29 10:49:27.284180 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:49:27.284130    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 29 10:49:37.284092 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:49:37.284042    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 29 10:49:47.284109 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:49:47.284069    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 29 10:49:57.284325 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:49:57.284282    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 29 10:50:07.284471 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:07.284427    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 29 10:50:17.294247 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:17.294216    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:50:17.302286 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:17.302253    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:50:29.145339 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:29.145307    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"]
Jul 29 10:50:44.166868 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.166825    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main" containerID="cri-o://99fc739175c4f2e22491a706fe9bbafde77b009139600288f624bbd6c0d8abfe" gracePeriod=45
Jul 29 10:50:44.295424 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.295387    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 29 10:50:44.911961 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.911934    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:50:44.980982 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.980899    2584 generic.go:356] "Generic (PLEG): container finished" podID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerID="99fc739175c4f2e22491a706fe9bbafde77b009139600288f624bbd6c0d8abfe" exitCode=0
Jul 29 10:50:44.980982 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.980951    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" event={"ID":"7778f8b5-6b3b-484c-8d88-0fac8e8242d3","Type":"ContainerDied","Data":"99fc739175c4f2e22491a706fe9bbafde77b009139600288f624bbd6c0d8abfe"}
Jul 29 10:50:44.980982 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.980953    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"
Jul 29 10:50:44.981224 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.980993    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx" event={"ID":"7778f8b5-6b3b-484c-8d88-0fac8e8242d3","Type":"ContainerDied","Data":"93097be976c5df8de2b4294aff3733404ca9d6de59d81555b8df924932f7d550"}
Jul 29 10:50:44.981224 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.981015    2584 scope.go:117] "RemoveContainer" containerID="99fc739175c4f2e22491a706fe9bbafde77b009139600288f624bbd6c0d8abfe"
Jul 29 10:50:44.989470 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.989446    2584 scope.go:117] "RemoveContainer" containerID="43eb6f3b6965b658184e3dfae9ede25eaa230ac55dd0c0193af435163bfeb5a6"
Jul 29 10:50:44.990396 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.990380    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-model-cache\") pod \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") "
Jul 29 10:50:44.990460 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.990419    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-home\") pod \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") "
Jul 29 10:50:44.990524 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.990489    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-tls-certs\") pod \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") "
Jul 29 10:50:44.990584 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.990531    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-tmp-dir\") pod \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") "
Jul 29 10:50:44.990584 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.990564    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bcsp\" (UniqueName: \"kubernetes.io/projected/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-kube-api-access-2bcsp\") pod \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") "
Jul 29 10:50:44.991101 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.990587    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-kserve-provision-location\") pod \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") "
Jul 29 10:50:44.991101 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.990619    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-dshm\") pod \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\" (UID: \"7778f8b5-6b3b-484c-8d88-0fac8e8242d3\") "
Jul 29 10:50:44.991101 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.990717    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "7778f8b5-6b3b-484c-8d88-0fac8e8242d3" (UID: "7778f8b5-6b3b-484c-8d88-0fac8e8242d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:50:44.991101 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.990938    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-model-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:50:44.991312 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.991120    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-home" (OuterVolumeSpecName: "home") pod "7778f8b5-6b3b-484c-8d88-0fac8e8242d3" (UID: "7778f8b5-6b3b-484c-8d88-0fac8e8242d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:50:44.993035 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.993000    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-kube-api-access-2bcsp" (OuterVolumeSpecName: "kube-api-access-2bcsp") pod "7778f8b5-6b3b-484c-8d88-0fac8e8242d3" (UID: "7778f8b5-6b3b-484c-8d88-0fac8e8242d3"). InnerVolumeSpecName "kube-api-access-2bcsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:50:44.993456 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.993429    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-dshm" (OuterVolumeSpecName: "dshm") pod "7778f8b5-6b3b-484c-8d88-0fac8e8242d3" (UID: "7778f8b5-6b3b-484c-8d88-0fac8e8242d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:50:44.993554 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:44.993519    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7778f8b5-6b3b-484c-8d88-0fac8e8242d3" (UID: "7778f8b5-6b3b-484c-8d88-0fac8e8242d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:50:45.002403 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:45.002373    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7778f8b5-6b3b-484c-8d88-0fac8e8242d3" (UID: "7778f8b5-6b3b-484c-8d88-0fac8e8242d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:50:45.010271 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:45.010249    2584 scope.go:117] "RemoveContainer" containerID="99fc739175c4f2e22491a706fe9bbafde77b009139600288f624bbd6c0d8abfe"
Jul 29 10:50:45.010571 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:50:45.010554    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99fc739175c4f2e22491a706fe9bbafde77b009139600288f624bbd6c0d8abfe\": container with ID starting with 99fc739175c4f2e22491a706fe9bbafde77b009139600288f624bbd6c0d8abfe not found: ID does not exist" containerID="99fc739175c4f2e22491a706fe9bbafde77b009139600288f624bbd6c0d8abfe"
Jul 29 10:50:45.010715 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:45.010580    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fc739175c4f2e22491a706fe9bbafde77b009139600288f624bbd6c0d8abfe"} err="failed to get container status \"99fc739175c4f2e22491a706fe9bbafde77b009139600288f624bbd6c0d8abfe\": rpc error: code = NotFound desc = could not find container \"99fc739175c4f2e22491a706fe9bbafde77b009139600288f624bbd6c0d8abfe\": container with ID starting with 99fc739175c4f2e22491a706fe9bbafde77b009139600288f624bbd6c0d8abfe not found: ID does not exist"
Jul 29 10:50:45.010715 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:45.010599    2584 scope.go:117] "RemoveContainer" containerID="43eb6f3b6965b658184e3dfae9ede25eaa230ac55dd0c0193af435163bfeb5a6"
Jul 29 10:50:45.010889 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:50:45.010871    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43eb6f3b6965b658184e3dfae9ede25eaa230ac55dd0c0193af435163bfeb5a6\": container with ID starting with 43eb6f3b6965b658184e3dfae9ede25eaa230ac55dd0c0193af435163bfeb5a6 not found: ID does not exist" containerID="43eb6f3b6965b658184e3dfae9ede25eaa230ac55dd0c0193af435163bfeb5a6"
Jul 29 10:50:45.010948 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:45.010896    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43eb6f3b6965b658184e3dfae9ede25eaa230ac55dd0c0193af435163bfeb5a6"} err="failed to get container status \"43eb6f3b6965b658184e3dfae9ede25eaa230ac55dd0c0193af435163bfeb5a6\": rpc error: code = NotFound desc = could not find container \"43eb6f3b6965b658184e3dfae9ede25eaa230ac55dd0c0193af435163bfeb5a6\": container with ID starting with 43eb6f3b6965b658184e3dfae9ede25eaa230ac55dd0c0193af435163bfeb5a6 not found: ID does not exist"
Jul 29 10:50:45.055632 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:45.055587    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7778f8b5-6b3b-484c-8d88-0fac8e8242d3" (UID: "7778f8b5-6b3b-484c-8d88-0fac8e8242d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:50:45.091719 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:45.091668    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-dshm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:50:45.091719 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:45.091714    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-home\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:50:45.091898 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:45.091727    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:50:45.091898 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:45.091740    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:50:45.091898 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:45.091752    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bcsp\" (UniqueName: \"kubernetes.io/projected/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-kube-api-access-2bcsp\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:50:45.091898 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:45.091763    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7778f8b5-6b3b-484c-8d88-0fac8e8242d3-kserve-provision-location\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:50:45.305388 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:45.305356    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"]
Jul 29 10:50:45.308634 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:45.308611    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6977dd4757-nxfxx"]
Jul 29 10:50:46.255259 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:50:46.255218    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" path="/var/lib/kubelet/pods/7778f8b5-6b3b-484c-8d88-0fac8e8242d3/volumes"
Jul 29 10:51:02.521623 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.521589    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"]
Jul 29 10:51:02.522062 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.521983    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="storage-initializer"
Jul 29 10:51:02.522062 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.521995    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="storage-initializer"
Jul 29 10:51:02.522062 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.522011    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main"
Jul 29 10:51:02.522062 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.522017    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main"
Jul 29 10:51:02.522203 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.522070    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7778f8b5-6b3b-484c-8d88-0fac8e8242d3" containerName="main"
Jul 29 10:51:02.530365 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.530336    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.533418 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.533395    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 29 10:51:02.533593 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.533460    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Jul 29 10:51:02.534815 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.534797    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-s526k\""
Jul 29 10:51:02.534921 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.534796    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 29 10:51:02.535737 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.535717    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 29 10:51:02.538194 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.538174    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"]
Jul 29 10:51:02.649919 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.649890    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7a1bc4-3368-4b06-bcd8-871b98510a88-tls-certs\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.650101 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.649926    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-dshm\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.650101 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.649950    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-model-cache\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.650101 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.650042    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-kserve-provision-location\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.650101 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.650089    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-tmp-dir\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.650267 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.650123    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhvnb\" (UniqueName: \"kubernetes.io/projected/6c7a1bc4-3368-4b06-bcd8-871b98510a88-kube-api-access-mhvnb\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.650267 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.650151    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-home\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.750974 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.750942    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-tmp-dir\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.751145 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.750983    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhvnb\" (UniqueName: \"kubernetes.io/projected/6c7a1bc4-3368-4b06-bcd8-871b98510a88-kube-api-access-mhvnb\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.751145 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.751014    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-home\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.751145 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.751069    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7a1bc4-3368-4b06-bcd8-871b98510a88-tls-certs\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.751291 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.751267    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-dshm\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.751340 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.751306    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-model-cache\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.751399 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.751350    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-kserve-provision-location\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.751399 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.751355    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-tmp-dir\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.751399 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.751395    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-home\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.751626 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.751600    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-model-cache\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.751781 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.751652    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-kserve-provision-location\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.753418 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.753396    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-dshm\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.753727 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.753707    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7a1bc4-3368-4b06-bcd8-871b98510a88-tls-certs\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.763301 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.763274    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhvnb\" (UniqueName: \"kubernetes.io/projected/6c7a1bc4-3368-4b06-bcd8-871b98510a88-kube-api-access-mhvnb\") pod \"auth-disabled-test-kserve-848c95d75d-hs4mj\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.841165 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.841127    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:02.976811 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:02.976786    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"]
Jul 29 10:51:02.978746 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:51:02.978720    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7a1bc4_3368_4b06_bcd8_871b98510a88.slice/crio-b3e4f7358febaa67b5b8706dc52178110e337dc9e03fc85737f705fa61ce0584 WatchSource:0}: Error finding container b3e4f7358febaa67b5b8706dc52178110e337dc9e03fc85737f705fa61ce0584: Status 404 returned error can't find the container with id b3e4f7358febaa67b5b8706dc52178110e337dc9e03fc85737f705fa61ce0584
Jul 29 10:51:03.065212 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:03.065174    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" event={"ID":"6c7a1bc4-3368-4b06-bcd8-871b98510a88","Type":"ContainerStarted","Data":"d5262de1f01ad4751befffcacc7ab7c9f31aaccf5bb8fc43b963f3b0a57f710c"}
Jul 29 10:51:03.065382 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:03.065227    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" event={"ID":"6c7a1bc4-3368-4b06-bcd8-871b98510a88","Type":"ContainerStarted","Data":"b3e4f7358febaa67b5b8706dc52178110e337dc9e03fc85737f705fa61ce0584"}
Jul 29 10:51:08.092192 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:08.092152    2584 generic.go:356] "Generic (PLEG): container finished" podID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerID="d5262de1f01ad4751befffcacc7ab7c9f31aaccf5bb8fc43b963f3b0a57f710c" exitCode=0
Jul 29 10:51:08.092617 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:08.092221    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" event={"ID":"6c7a1bc4-3368-4b06-bcd8-871b98510a88","Type":"ContainerDied","Data":"d5262de1f01ad4751befffcacc7ab7c9f31aaccf5bb8fc43b963f3b0a57f710c"}
Jul 29 10:51:09.098774 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:09.098738    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" event={"ID":"6c7a1bc4-3368-4b06-bcd8-871b98510a88","Type":"ContainerStarted","Data":"bce611e0ee8bf3b0c79a1b6d1dcf4a6fb37621c931ccc7d6c3a4cc0409ae4a3d"}
Jul 29 10:51:09.124400 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:09.124347    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" podStartSLOduration=2.282411138 podStartE2EDuration="7.124329319s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.841918181s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:51:09.122827889 +0000 UTC m=+1337.315855239" watchObservedRunningTime="2026-07-29 10:51:09.124329319 +0000 UTC m=+1337.317356667"
Jul 29 10:51:09.842228 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:09.842200    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:12.842188 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:12.842151    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:51:12.844152 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:12.844119    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 29 10:51:22.842236 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:22.842128    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 29 10:51:32.842455 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:32.842360    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 29 10:51:36.627107 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:36.627070    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"]
Jul 29 10:51:36.629698 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:36.629662    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"
Jul 29 10:51:36.634602 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:36.634444    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 29 10:51:36.634602 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:36.634444    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-c8zd7\""
Jul 29 10:51:36.634602 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:36.634467    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 29 10:51:36.634602 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:36.634503    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 29 10:51:36.634909 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:36.634776    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-k6qbt\""
Jul 29 10:51:36.645097 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:36.645031    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"]
Jul 29 10:51:36.740889 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:36.740852    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630fff6b-adb0-4e3c-98a6-08350137f7ae-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5\" (UID: \"630fff6b-adb0-4e3c-98a6-08350137f7ae\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"
Jul 29 10:51:36.741087 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:36.740915    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkk8j\" (UniqueName: \"kubernetes.io/projected/630fff6b-adb0-4e3c-98a6-08350137f7ae-kube-api-access-gkk8j\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5\" (UID: \"630fff6b-adb0-4e3c-98a6-08350137f7ae\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"
Jul 29 10:51:36.842142 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:36.842098    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkk8j\" (UniqueName: \"kubernetes.io/projected/630fff6b-adb0-4e3c-98a6-08350137f7ae-kube-api-access-gkk8j\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5\" (UID: \"630fff6b-adb0-4e3c-98a6-08350137f7ae\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"
Jul 29 10:51:36.842346 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:36.842169    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630fff6b-adb0-4e3c-98a6-08350137f7ae-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5\" (UID: \"630fff6b-adb0-4e3c-98a6-08350137f7ae\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"
Jul 29 10:51:36.844723 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:36.844667    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630fff6b-adb0-4e3c-98a6-08350137f7ae-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5\" (UID: \"630fff6b-adb0-4e3c-98a6-08350137f7ae\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"
Jul 29 10:51:36.851757 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:36.851733    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkk8j\" (UniqueName: \"kubernetes.io/projected/630fff6b-adb0-4e3c-98a6-08350137f7ae-kube-api-access-gkk8j\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5\" (UID: \"630fff6b-adb0-4e3c-98a6-08350137f7ae\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"
Jul 29 10:51:36.943773 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:36.943668    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"
Jul 29 10:51:37.076990 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:37.076967    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"]
Jul 29 10:51:37.078989 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:51:37.078966    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630fff6b_adb0_4e3c_98a6_08350137f7ae.slice/crio-ba51621ca1cc79db6406c3d12d2670477ce29d0098efacf553c5de7855020240 WatchSource:0}: Error finding container ba51621ca1cc79db6406c3d12d2670477ce29d0098efacf553c5de7855020240: Status 404 returned error can't find the container with id ba51621ca1cc79db6406c3d12d2670477ce29d0098efacf553c5de7855020240
Jul 29 10:51:37.080910 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:37.080893    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:51:37.232010 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:37.231920    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5" event={"ID":"630fff6b-adb0-4e3c-98a6-08350137f7ae","Type":"ContainerStarted","Data":"ba51621ca1cc79db6406c3d12d2670477ce29d0098efacf553c5de7855020240"}
Jul 29 10:51:40.248713 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:40.248655    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5" event={"ID":"630fff6b-adb0-4e3c-98a6-08350137f7ae","Type":"ContainerStarted","Data":"c8c3888fc7fb7c186e9e8236e51e3d11857211ba178739d5a5d37e7a6f1094f1"}
Jul 29 10:51:40.249219 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:40.248784    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"
Jul 29 10:51:40.266300 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:40.266007    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5" podStartSLOduration=1.80176415 podStartE2EDuration="4.265984694s" totalImagesPullingTime="2.464220544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:51:40.265152948 +0000 UTC m=+1368.458180297" watchObservedRunningTime="2026-07-29 10:51:40.265984694 +0000 UTC m=+1368.459012042"
Jul 29 10:51:42.841793 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:42.841742    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 29 10:51:52.338442 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:52.338409    2584 scope.go:117] "RemoveContainer" containerID="496602db655dda1ed0a1e48ac3de2894a4adf98db942a987d300b88d621cb12f"
Jul 29 10:51:52.346480 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:52.346451    2584 scope.go:117] "RemoveContainer" containerID="3de9c9b999d7ad5ad271f901a972136d8326fee9888fbb505c50ed89646745f4"
Jul 29 10:51:52.841773 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:51:52.841732    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 29 10:52:02.842009 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:02.841965    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 29 10:52:11.255166 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:11.255128    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5" podUID="630fff6b-adb0-4e3c-98a6-08350137f7ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:52:12.473082 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:12.473045    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"]
Jul 29 10:52:12.474795 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:12.474761    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5" podUID="630fff6b-adb0-4e3c-98a6-08350137f7ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:52:12.842522 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:12.842481    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 29 10:52:22.475068 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:22.475032    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5" podUID="630fff6b-adb0-4e3c-98a6-08350137f7ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:52:22.842420 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:22.842379    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 29 10:52:27.492412 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:27.492377    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5" podUID="630fff6b-adb0-4e3c-98a6-08350137f7ae" containerName="main" containerID="cri-o://c8c3888fc7fb7c186e9e8236e51e3d11857211ba178739d5a5d37e7a6f1094f1" gracePeriod=45
Jul 29 10:52:27.735945 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:27.735923    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"
Jul 29 10:52:27.889739 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:27.889706    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkk8j\" (UniqueName: \"kubernetes.io/projected/630fff6b-adb0-4e3c-98a6-08350137f7ae-kube-api-access-gkk8j\") pod \"630fff6b-adb0-4e3c-98a6-08350137f7ae\" (UID: \"630fff6b-adb0-4e3c-98a6-08350137f7ae\") "
Jul 29 10:52:27.889927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:27.889834    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630fff6b-adb0-4e3c-98a6-08350137f7ae-tls-certs\") pod \"630fff6b-adb0-4e3c-98a6-08350137f7ae\" (UID: \"630fff6b-adb0-4e3c-98a6-08350137f7ae\") "
Jul 29 10:52:27.891932 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:27.891905    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630fff6b-adb0-4e3c-98a6-08350137f7ae-kube-api-access-gkk8j" (OuterVolumeSpecName: "kube-api-access-gkk8j") pod "630fff6b-adb0-4e3c-98a6-08350137f7ae" (UID: "630fff6b-adb0-4e3c-98a6-08350137f7ae"). InnerVolumeSpecName "kube-api-access-gkk8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:52:27.892056 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:27.891948    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630fff6b-adb0-4e3c-98a6-08350137f7ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "630fff6b-adb0-4e3c-98a6-08350137f7ae" (UID: "630fff6b-adb0-4e3c-98a6-08350137f7ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:52:27.991080 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:27.991048    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630fff6b-adb0-4e3c-98a6-08350137f7ae-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:52:27.991080 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:27.991074    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkk8j\" (UniqueName: \"kubernetes.io/projected/630fff6b-adb0-4e3c-98a6-08350137f7ae-kube-api-access-gkk8j\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:52:28.473990 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:28.473961    2584 generic.go:356] "Generic (PLEG): container finished" podID="630fff6b-adb0-4e3c-98a6-08350137f7ae" containerID="c8c3888fc7fb7c186e9e8236e51e3d11857211ba178739d5a5d37e7a6f1094f1" exitCode=0
Jul 29 10:52:28.474170 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:28.474008    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"
Jul 29 10:52:28.474170 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:28.474028    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5" event={"ID":"630fff6b-adb0-4e3c-98a6-08350137f7ae","Type":"ContainerDied","Data":"c8c3888fc7fb7c186e9e8236e51e3d11857211ba178739d5a5d37e7a6f1094f1"}
Jul 29 10:52:28.474170 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:28.474069    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5" event={"ID":"630fff6b-adb0-4e3c-98a6-08350137f7ae","Type":"ContainerDied","Data":"ba51621ca1cc79db6406c3d12d2670477ce29d0098efacf553c5de7855020240"}
Jul 29 10:52:28.474170 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:28.474091    2584 scope.go:117] "RemoveContainer" containerID="c8c3888fc7fb7c186e9e8236e51e3d11857211ba178739d5a5d37e7a6f1094f1"
Jul 29 10:52:28.483951 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:28.483926    2584 scope.go:117] "RemoveContainer" containerID="c8c3888fc7fb7c186e9e8236e51e3d11857211ba178739d5a5d37e7a6f1094f1"
Jul 29 10:52:28.484223 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:52:28.484205    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c3888fc7fb7c186e9e8236e51e3d11857211ba178739d5a5d37e7a6f1094f1\": container with ID starting with c8c3888fc7fb7c186e9e8236e51e3d11857211ba178739d5a5d37e7a6f1094f1 not found: ID does not exist" containerID="c8c3888fc7fb7c186e9e8236e51e3d11857211ba178739d5a5d37e7a6f1094f1"
Jul 29 10:52:28.484277 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:28.484233    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c3888fc7fb7c186e9e8236e51e3d11857211ba178739d5a5d37e7a6f1094f1"} err="failed to get container status \"c8c3888fc7fb7c186e9e8236e51e3d11857211ba178739d5a5d37e7a6f1094f1\": rpc error: code = NotFound desc = could not find container \"c8c3888fc7fb7c186e9e8236e51e3d11857211ba178739d5a5d37e7a6f1094f1\": container with ID starting with c8c3888fc7fb7c186e9e8236e51e3d11857211ba178739d5a5d37e7a6f1094f1 not found: ID does not exist"
Jul 29 10:52:28.491396 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:28.491376    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"]
Jul 29 10:52:28.497263 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:28.497240    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf55sg5"]
Jul 29 10:52:30.260373 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:30.260334    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="630fff6b-adb0-4e3c-98a6-08350137f7ae" path="/var/lib/kubelet/pods/630fff6b-adb0-4e3c-98a6-08350137f7ae/volumes"
Jul 29 10:52:32.842445 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:32.842400    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 29 10:52:39.777858 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.777824    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"]
Jul 29 10:52:39.778421 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.778397    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="630fff6b-adb0-4e3c-98a6-08350137f7ae" containerName="main"
Jul 29 10:52:39.778503 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.778426    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="630fff6b-adb0-4e3c-98a6-08350137f7ae" containerName="main"
Jul 29 10:52:39.778555 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.778516    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="630fff6b-adb0-4e3c-98a6-08350137f7ae" containerName="main"
Jul 29 10:52:39.782794 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.782773    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:52:39.787115 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.787079    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 29 10:52:39.787259 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.787152    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 29 10:52:39.787259 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.787203    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-p9kb4\""
Jul 29 10:52:39.787259 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.787206    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-rg7g2\""
Jul 29 10:52:39.787420 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.787309    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 29 10:52:39.793175 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.793153    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"]
Jul 29 10:52:39.893176 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.893138    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795qmghs\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:52:39.893357 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.893194    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795qmghs\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:52:39.893357 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.893265    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckcqp\" (UniqueName: \"kubernetes.io/projected/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-kube-api-access-ckcqp\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795qmghs\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:52:39.893357 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.893325    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795qmghs\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:52:39.994127 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.994089    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795qmghs\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:52:39.994323 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.994167    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795qmghs\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:52:39.994323 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.994209    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795qmghs\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:52:39.994323 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.994240    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckcqp\" (UniqueName: \"kubernetes.io/projected/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-kube-api-access-ckcqp\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795qmghs\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:52:39.994773 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.994546    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795qmghs\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:52:39.994773 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.994604    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795qmghs\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:52:39.996700 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:39.996656    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795qmghs\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:52:40.002515 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:40.002489    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckcqp\" (UniqueName: \"kubernetes.io/projected/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-kube-api-access-ckcqp\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795qmghs\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:52:40.095714 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:40.095671    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:52:40.238973 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:40.238951    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"]
Jul 29 10:52:40.241211 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:52:40.241173    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8f8139_aeaa_4cec_8599_2e7de4dd9fe2.slice/crio-3f6f920fcbbbd9ae8751380c4af9bc566ff7c736e695f7f8a20c1d3577a3ea52 WatchSource:0}: Error finding container 3f6f920fcbbbd9ae8751380c4af9bc566ff7c736e695f7f8a20c1d3577a3ea52: Status 404 returned error can't find the container with id 3f6f920fcbbbd9ae8751380c4af9bc566ff7c736e695f7f8a20c1d3577a3ea52
Jul 29 10:52:40.537880 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:40.537845    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs" event={"ID":"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2","Type":"ContainerStarted","Data":"afe0c580bb3c4a513716c64c73bfe67a71017a94e0f1b43fd6279a36d0e52148"}
Jul 29 10:52:40.538084 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:40.537910    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs" event={"ID":"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2","Type":"ContainerStarted","Data":"3f6f920fcbbbd9ae8751380c4af9bc566ff7c736e695f7f8a20c1d3577a3ea52"}
Jul 29 10:52:40.538348 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:40.538325    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:52:40.559733 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:40.559656    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs" podStartSLOduration=1.559635794 podStartE2EDuration="1.559635794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:52:40.558912457 +0000 UTC m=+1428.751939804" watchObservedRunningTime="2026-07-29 10:52:40.559635794 +0000 UTC m=+1428.752663142"
Jul 29 10:52:42.841804 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:42.841754    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 29 10:52:52.842040 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:52:52.841999    2584 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 29 10:53:02.851845 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:02.851815    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:53:02.859814 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:02.859792    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:53:06.926974 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:06.926930    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"]
Jul 29 10:53:11.545328 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:11.545285    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:53:12.760637 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:12.760603    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"]
Jul 29 10:53:21.545175 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:21.545130    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs" podUID="7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:53:21.949565 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:21.949475    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="main" containerID="cri-o://bce611e0ee8bf3b0c79a1b6d1dcf4a6fb37621c931ccc7d6c3a4cc0409ae4a3d" gracePeriod=45
Jul 29 10:53:22.694518 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.694492    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:53:22.732033 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.731952    2584 generic.go:356] "Generic (PLEG): container finished" podID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerID="bce611e0ee8bf3b0c79a1b6d1dcf4a6fb37621c931ccc7d6c3a4cc0409ae4a3d" exitCode=0
Jul 29 10:53:22.732033 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.732000    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"
Jul 29 10:53:22.732260 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.732033    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" event={"ID":"6c7a1bc4-3368-4b06-bcd8-871b98510a88","Type":"ContainerDied","Data":"bce611e0ee8bf3b0c79a1b6d1dcf4a6fb37621c931ccc7d6c3a4cc0409ae4a3d"}
Jul 29 10:53:22.732260 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.732088    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj" event={"ID":"6c7a1bc4-3368-4b06-bcd8-871b98510a88","Type":"ContainerDied","Data":"b3e4f7358febaa67b5b8706dc52178110e337dc9e03fc85737f705fa61ce0584"}
Jul 29 10:53:22.732260 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.732111    2584 scope.go:117] "RemoveContainer" containerID="bce611e0ee8bf3b0c79a1b6d1dcf4a6fb37621c931ccc7d6c3a4cc0409ae4a3d"
Jul 29 10:53:22.741266 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.741250    2584 scope.go:117] "RemoveContainer" containerID="d5262de1f01ad4751befffcacc7ab7c9f31aaccf5bb8fc43b963f3b0a57f710c"
Jul 29 10:53:22.749665 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.749641    2584 scope.go:117] "RemoveContainer" containerID="bce611e0ee8bf3b0c79a1b6d1dcf4a6fb37621c931ccc7d6c3a4cc0409ae4a3d"
Jul 29 10:53:22.749964 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:53:22.749943    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce611e0ee8bf3b0c79a1b6d1dcf4a6fb37621c931ccc7d6c3a4cc0409ae4a3d\": container with ID starting with bce611e0ee8bf3b0c79a1b6d1dcf4a6fb37621c931ccc7d6c3a4cc0409ae4a3d not found: ID does not exist" containerID="bce611e0ee8bf3b0c79a1b6d1dcf4a6fb37621c931ccc7d6c3a4cc0409ae4a3d"
Jul 29 10:53:22.750017 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.749970    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce611e0ee8bf3b0c79a1b6d1dcf4a6fb37621c931ccc7d6c3a4cc0409ae4a3d"} err="failed to get container status \"bce611e0ee8bf3b0c79a1b6d1dcf4a6fb37621c931ccc7d6c3a4cc0409ae4a3d\": rpc error: code = NotFound desc = could not find container \"bce611e0ee8bf3b0c79a1b6d1dcf4a6fb37621c931ccc7d6c3a4cc0409ae4a3d\": container with ID starting with bce611e0ee8bf3b0c79a1b6d1dcf4a6fb37621c931ccc7d6c3a4cc0409ae4a3d not found: ID does not exist"
Jul 29 10:53:22.750017 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.749989    2584 scope.go:117] "RemoveContainer" containerID="d5262de1f01ad4751befffcacc7ab7c9f31aaccf5bb8fc43b963f3b0a57f710c"
Jul 29 10:53:22.750231 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:53:22.750215    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5262de1f01ad4751befffcacc7ab7c9f31aaccf5bb8fc43b963f3b0a57f710c\": container with ID starting with d5262de1f01ad4751befffcacc7ab7c9f31aaccf5bb8fc43b963f3b0a57f710c not found: ID does not exist" containerID="d5262de1f01ad4751befffcacc7ab7c9f31aaccf5bb8fc43b963f3b0a57f710c"
Jul 29 10:53:22.750280 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.750238    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5262de1f01ad4751befffcacc7ab7c9f31aaccf5bb8fc43b963f3b0a57f710c"} err="failed to get container status \"d5262de1f01ad4751befffcacc7ab7c9f31aaccf5bb8fc43b963f3b0a57f710c\": rpc error: code = NotFound desc = could not find container \"d5262de1f01ad4751befffcacc7ab7c9f31aaccf5bb8fc43b963f3b0a57f710c\": container with ID starting with d5262de1f01ad4751befffcacc7ab7c9f31aaccf5bb8fc43b963f3b0a57f710c not found: ID does not exist"
Jul 29 10:53:22.778542 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.778518    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-dshm\") pod \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") "
Jul 29 10:53:22.778671 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.778570    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7a1bc4-3368-4b06-bcd8-871b98510a88-tls-certs\") pod \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") "
Jul 29 10:53:22.778671 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.778593    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhvnb\" (UniqueName: \"kubernetes.io/projected/6c7a1bc4-3368-4b06-bcd8-871b98510a88-kube-api-access-mhvnb\") pod \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") "
Jul 29 10:53:22.778811 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.778750    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-model-cache\") pod \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") "
Jul 29 10:53:22.778811 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.778800    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-kserve-provision-location\") pod \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") "
Jul 29 10:53:22.778915 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.778831    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-tmp-dir\") pod \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") "
Jul 29 10:53:22.778915 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.778868    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-home\") pod \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\" (UID: \"6c7a1bc4-3368-4b06-bcd8-871b98510a88\") "
Jul 29 10:53:22.779187 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.779052    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c7a1bc4-3368-4b06-bcd8-871b98510a88" (UID: "6c7a1bc4-3368-4b06-bcd8-871b98510a88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:53:22.779298 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.779209    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-model-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:53:22.779665 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.779642    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-home" (OuterVolumeSpecName: "home") pod "6c7a1bc4-3368-4b06-bcd8-871b98510a88" (UID: "6c7a1bc4-3368-4b06-bcd8-871b98510a88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:53:22.781033 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.781008    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7a1bc4-3368-4b06-bcd8-871b98510a88-kube-api-access-mhvnb" (OuterVolumeSpecName: "kube-api-access-mhvnb") pod "6c7a1bc4-3368-4b06-bcd8-871b98510a88" (UID: "6c7a1bc4-3368-4b06-bcd8-871b98510a88"). InnerVolumeSpecName "kube-api-access-mhvnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:53:22.781137 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.781052    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-dshm" (OuterVolumeSpecName: "dshm") pod "6c7a1bc4-3368-4b06-bcd8-871b98510a88" (UID: "6c7a1bc4-3368-4b06-bcd8-871b98510a88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:53:22.781137 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.781129    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c7a1bc4-3368-4b06-bcd8-871b98510a88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c7a1bc4-3368-4b06-bcd8-871b98510a88" (UID: "6c7a1bc4-3368-4b06-bcd8-871b98510a88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:53:22.791712 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.791663    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c7a1bc4-3368-4b06-bcd8-871b98510a88" (UID: "6c7a1bc4-3368-4b06-bcd8-871b98510a88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:53:22.844390 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.844352    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c7a1bc4-3368-4b06-bcd8-871b98510a88" (UID: "6c7a1bc4-3368-4b06-bcd8-871b98510a88"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:53:22.879671 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.879640    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-kserve-provision-location\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:53:22.879671 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.879669    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:53:22.879671 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.879694    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-home\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:53:22.879671 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.879701    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c7a1bc4-3368-4b06-bcd8-871b98510a88-dshm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:53:22.879929 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.879710    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7a1bc4-3368-4b06-bcd8-871b98510a88-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:53:22.879929 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:22.879717    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhvnb\" (UniqueName: \"kubernetes.io/projected/6c7a1bc4-3368-4b06-bcd8-871b98510a88-kube-api-access-mhvnb\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:53:23.055242 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:23.055216    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"]
Jul 29 10:53:23.059155 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:23.059128    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-848c95d75d-hs4mj"]
Jul 29 10:53:24.254877 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:24.254846    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" path="/var/lib/kubelet/pods/6c7a1bc4-3368-4b06-bcd8-871b98510a88/volumes"
Jul 29 10:53:27.779600 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:27.779556    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs" podUID="7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2" containerName="main" containerID="cri-o://afe0c580bb3c4a513716c64c73bfe67a71017a94e0f1b43fd6279a36d0e52148" gracePeriod=45
Jul 29 10:53:28.030970 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.030918    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:53:28.124844 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.124805    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-tls-certs\") pod \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") "
Jul 29 10:53:28.125040 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.124896    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-cachi2-cache\") pod \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") "
Jul 29 10:53:28.125040 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.124980    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckcqp\" (UniqueName: \"kubernetes.io/projected/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-kube-api-access-ckcqp\") pod \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") "
Jul 29 10:53:28.125040 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.125006    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-tokenizer-cache\") pod \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\" (UID: \"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2\") "
Jul 29 10:53:28.125229 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.125198    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2" (UID: "7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:53:28.125352 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.125333    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2" (UID: "7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:53:28.127067 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.127047    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-kube-api-access-ckcqp" (OuterVolumeSpecName: "kube-api-access-ckcqp") pod "7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2" (UID: "7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2"). InnerVolumeSpecName "kube-api-access-ckcqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:53:28.127067 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.127057    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2" (UID: "7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:53:28.226668 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.226626    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckcqp\" (UniqueName: \"kubernetes.io/projected/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-kube-api-access-ckcqp\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:53:28.226668 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.226661    2584 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-tokenizer-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:53:28.226668 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.226671    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:53:28.226668 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.226705    2584 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2-cachi2-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:53:28.757911 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.757872    2584 generic.go:356] "Generic (PLEG): container finished" podID="7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2" containerID="afe0c580bb3c4a513716c64c73bfe67a71017a94e0f1b43fd6279a36d0e52148" exitCode=0
Jul 29 10:53:28.758092 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.757934    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"
Jul 29 10:53:28.758092 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.757960    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs" event={"ID":"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2","Type":"ContainerDied","Data":"afe0c580bb3c4a513716c64c73bfe67a71017a94e0f1b43fd6279a36d0e52148"}
Jul 29 10:53:28.758092 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.758026    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs" event={"ID":"7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2","Type":"ContainerDied","Data":"3f6f920fcbbbd9ae8751380c4af9bc566ff7c736e695f7f8a20c1d3577a3ea52"}
Jul 29 10:53:28.758092 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.758055    2584 scope.go:117] "RemoveContainer" containerID="afe0c580bb3c4a513716c64c73bfe67a71017a94e0f1b43fd6279a36d0e52148"
Jul 29 10:53:28.766151 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.766128    2584 scope.go:117] "RemoveContainer" containerID="afe0c580bb3c4a513716c64c73bfe67a71017a94e0f1b43fd6279a36d0e52148"
Jul 29 10:53:28.766429 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:53:28.766409    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe0c580bb3c4a513716c64c73bfe67a71017a94e0f1b43fd6279a36d0e52148\": container with ID starting with afe0c580bb3c4a513716c64c73bfe67a71017a94e0f1b43fd6279a36d0e52148 not found: ID does not exist" containerID="afe0c580bb3c4a513716c64c73bfe67a71017a94e0f1b43fd6279a36d0e52148"
Jul 29 10:53:28.766484 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.766437    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe0c580bb3c4a513716c64c73bfe67a71017a94e0f1b43fd6279a36d0e52148"} err="failed to get container status \"afe0c580bb3c4a513716c64c73bfe67a71017a94e0f1b43fd6279a36d0e52148\": rpc error: code = NotFound desc = could not find container \"afe0c580bb3c4a513716c64c73bfe67a71017a94e0f1b43fd6279a36d0e52148\": container with ID starting with afe0c580bb3c4a513716c64c73bfe67a71017a94e0f1b43fd6279a36d0e52148 not found: ID does not exist"
Jul 29 10:53:28.778279 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.778253    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"]
Jul 29 10:53:28.780268 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:28.780246    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795qmghs"]
Jul 29 10:53:30.255252 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:30.255208    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2" path="/var/lib/kubelet/pods/7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2/volumes"
Jul 29 10:53:31.133959 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.133929    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"]
Jul 29 10:53:31.134248 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.134236    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="main"
Jul 29 10:53:31.134302 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.134250    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="main"
Jul 29 10:53:31.134302 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.134287    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="storage-initializer"
Jul 29 10:53:31.134302 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.134293    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="storage-initializer"
Jul 29 10:53:31.134302 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.134300    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2" containerName="main"
Jul 29 10:53:31.134430 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.134306    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2" containerName="main"
Jul 29 10:53:31.134430 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.134364    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c7a1bc4-3368-4b06-bcd8-871b98510a88" containerName="main"
Jul 29 10:53:31.134430 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.134373    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a8f8139-aeaa-4cec-8599-2e7de4dd9fe2" containerName="main"
Jul 29 10:53:31.138809 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.138792    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.141883 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.141860    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"openshift-service-ca.crt\""
Jul 29 10:53:31.143250 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.143228    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"default-dockercfg-h5sj8\""
Jul 29 10:53:31.143372 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.143250    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"kube-root-ca.crt\""
Jul 29 10:53:31.143372 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.143322    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-kserve-self-signed-certs\""
Jul 29 10:53:31.146946 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.146921    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"]
Jul 29 10:53:31.253301 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.253259    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.253301 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.253304    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.253525 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.253370    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pndn\" (UniqueName: \"kubernetes.io/projected/7a9be8c2-2468-4164-a218-8e9322ae9db9-kube-api-access-2pndn\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.253525 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.253409    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.253525 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.253441    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a9be8c2-2468-4164-a218-8e9322ae9db9-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.253525 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.253459    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.354265 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.354224    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a9be8c2-2468-4164-a218-8e9322ae9db9-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.354265 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.354266    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.354785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.354366    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.354785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.354405    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.354785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.354461    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pndn\" (UniqueName: \"kubernetes.io/projected/7a9be8c2-2468-4164-a218-8e9322ae9db9-kube-api-access-2pndn\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.354785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.354490    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.354785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.354728    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.355031 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.354795    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.355394 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.355368    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.357223 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.357199    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a9be8c2-2468-4164-a218-8e9322ae9db9-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.357351 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.357332    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.366613 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.366587    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pndn\" (UniqueName: \"kubernetes.io/projected/7a9be8c2-2468-4164-a218-8e9322ae9db9-kube-api-access-2pndn\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.450006 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.449915    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:31.580420 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.580325    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"]
Jul 29 10:53:31.583226 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:53:31.583197    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9be8c2_2468_4164_a218_8e9322ae9db9.slice/crio-1acaf20d908acd9abf4465a4ebc3ca969c5cd9d9fcef2401505228266ad0a3ec WatchSource:0}: Error finding container 1acaf20d908acd9abf4465a4ebc3ca969c5cd9d9fcef2401505228266ad0a3ec: Status 404 returned error can't find the container with id 1acaf20d908acd9abf4465a4ebc3ca969c5cd9d9fcef2401505228266ad0a3ec
Jul 29 10:53:31.773610 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.773515    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k" event={"ID":"7a9be8c2-2468-4164-a218-8e9322ae9db9","Type":"ContainerStarted","Data":"ebffb5a0d76549b6cbe1f56c476b0f3c318cc6ef2689b17f21b7eef8078a4679"}
Jul 29 10:53:31.773610 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.773566    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k" event={"ID":"7a9be8c2-2468-4164-a218-8e9322ae9db9","Type":"ContainerStarted","Data":"1acaf20d908acd9abf4465a4ebc3ca969c5cd9d9fcef2401505228266ad0a3ec"}
Jul 29 10:53:31.791129 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:31.791081    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k" podStartSLOduration=0.791065339 podStartE2EDuration="791.065339ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:53:31.790776025 +0000 UTC m=+1479.983803374" watchObservedRunningTime="2026-07-29 10:53:31.791065339 +0000 UTC m=+1479.984092686"
Jul 29 10:53:32.450813 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:32.450760    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:41.450592 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:41.450552    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:41.462841 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:41.462816    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:53:41.827314 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:53:41.827285    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 10:54:03.360905 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.360864    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"]
Jul 29 10:54:03.365815 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.365795    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"
Jul 29 10:54:03.370366 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.370338    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-87z5c\""
Jul 29 10:54:03.370366 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.370354    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 29 10:54:03.370607 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.370384    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 29 10:54:03.370607 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.370420    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 29 10:54:03.370607 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.370338    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-jdw8n\""
Jul 29 10:54:03.375153 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.375132    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"]
Jul 29 10:54:03.436015 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.435976    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/625759cb-ae3d-45df-88d4-5948763e6a74-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb\" (UID: \"625759cb-ae3d-45df-88d4-5948763e6a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"
Jul 29 10:54:03.436015 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.436021    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hbsj\" (UniqueName: \"kubernetes.io/projected/625759cb-ae3d-45df-88d4-5948763e6a74-kube-api-access-9hbsj\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb\" (UID: \"625759cb-ae3d-45df-88d4-5948763e6a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"
Jul 29 10:54:03.537284 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.537250    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/625759cb-ae3d-45df-88d4-5948763e6a74-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb\" (UID: \"625759cb-ae3d-45df-88d4-5948763e6a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"
Jul 29 10:54:03.537284 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.537290    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hbsj\" (UniqueName: \"kubernetes.io/projected/625759cb-ae3d-45df-88d4-5948763e6a74-kube-api-access-9hbsj\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb\" (UID: \"625759cb-ae3d-45df-88d4-5948763e6a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"
Jul 29 10:54:03.539939 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.539910    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/625759cb-ae3d-45df-88d4-5948763e6a74-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb\" (UID: \"625759cb-ae3d-45df-88d4-5948763e6a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"
Jul 29 10:54:03.547748 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.547720    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hbsj\" (UniqueName: \"kubernetes.io/projected/625759cb-ae3d-45df-88d4-5948763e6a74-kube-api-access-9hbsj\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb\" (UID: \"625759cb-ae3d-45df-88d4-5948763e6a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"
Jul 29 10:54:03.677087 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.676986    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"
Jul 29 10:54:03.826232 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.826206    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"]
Jul 29 10:54:03.828195 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:54:03.828166    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625759cb_ae3d_45df_88d4_5948763e6a74.slice/crio-af6af7fbf55cb6013eca9c67deaaf3463f6e87b0e9bae4a49934611e43d82d13 WatchSource:0}: Error finding container af6af7fbf55cb6013eca9c67deaaf3463f6e87b0e9bae4a49934611e43d82d13: Status 404 returned error can't find the container with id af6af7fbf55cb6013eca9c67deaaf3463f6e87b0e9bae4a49934611e43d82d13
Jul 29 10:54:03.916336 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.916277    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb" event={"ID":"625759cb-ae3d-45df-88d4-5948763e6a74","Type":"ContainerStarted","Data":"c9081b2ee40e529e00707e3ed0aa648ff47d48b4540b97779e271e6785a1e4a7"}
Jul 29 10:54:03.916525 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.916352    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb" event={"ID":"625759cb-ae3d-45df-88d4-5948763e6a74","Type":"ContainerStarted","Data":"af6af7fbf55cb6013eca9c67deaaf3463f6e87b0e9bae4a49934611e43d82d13"}
Jul 29 10:54:03.916525 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.916415    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"
Jul 29 10:54:03.943208 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:03.943090    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb" podStartSLOduration=0.9430752 podStartE2EDuration="943.0752ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:54:03.942784773 +0000 UTC m=+1512.135812121" watchObservedRunningTime="2026-07-29 10:54:03.9430752 +0000 UTC m=+1512.136102546"
Jul 29 10:54:34.922986 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:54:34.922899    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"
Jul 29 10:55:36.360428 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:36.360395    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"]
Jul 29 10:55:44.922216 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:44.922125    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb" podUID="625759cb-ae3d-45df-88d4-5948763e6a74" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:55:51.379363 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:51.379323    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb" podUID="625759cb-ae3d-45df-88d4-5948763e6a74" containerName="main" containerID="cri-o://c9081b2ee40e529e00707e3ed0aa648ff47d48b4540b97779e271e6785a1e4a7" gracePeriod=45
Jul 29 10:55:51.636538 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:51.636476    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"
Jul 29 10:55:51.796016 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:51.795989    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hbsj\" (UniqueName: \"kubernetes.io/projected/625759cb-ae3d-45df-88d4-5948763e6a74-kube-api-access-9hbsj\") pod \"625759cb-ae3d-45df-88d4-5948763e6a74\" (UID: \"625759cb-ae3d-45df-88d4-5948763e6a74\") "
Jul 29 10:55:51.796184 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:51.796085    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/625759cb-ae3d-45df-88d4-5948763e6a74-tls-certs\") pod \"625759cb-ae3d-45df-88d4-5948763e6a74\" (UID: \"625759cb-ae3d-45df-88d4-5948763e6a74\") "
Jul 29 10:55:51.798220 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:51.798196    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625759cb-ae3d-45df-88d4-5948763e6a74-kube-api-access-9hbsj" (OuterVolumeSpecName: "kube-api-access-9hbsj") pod "625759cb-ae3d-45df-88d4-5948763e6a74" (UID: "625759cb-ae3d-45df-88d4-5948763e6a74"). InnerVolumeSpecName "kube-api-access-9hbsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:55:51.798299 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:51.798277    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625759cb-ae3d-45df-88d4-5948763e6a74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "625759cb-ae3d-45df-88d4-5948763e6a74" (UID: "625759cb-ae3d-45df-88d4-5948763e6a74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:55:51.897060 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:51.896973    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/625759cb-ae3d-45df-88d4-5948763e6a74-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:55:51.897060 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:51.897001    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hbsj\" (UniqueName: \"kubernetes.io/projected/625759cb-ae3d-45df-88d4-5948763e6a74-kube-api-access-9hbsj\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:55:52.407660 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:52.407627    2584 generic.go:356] "Generic (PLEG): container finished" podID="625759cb-ae3d-45df-88d4-5948763e6a74" containerID="c9081b2ee40e529e00707e3ed0aa648ff47d48b4540b97779e271e6785a1e4a7" exitCode=0
Jul 29 10:55:52.408130 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:52.407670    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"
Jul 29 10:55:52.408130 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:52.407698    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb" event={"ID":"625759cb-ae3d-45df-88d4-5948763e6a74","Type":"ContainerDied","Data":"c9081b2ee40e529e00707e3ed0aa648ff47d48b4540b97779e271e6785a1e4a7"}
Jul 29 10:55:52.408130 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:52.407748    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb" event={"ID":"625759cb-ae3d-45df-88d4-5948763e6a74","Type":"ContainerDied","Data":"af6af7fbf55cb6013eca9c67deaaf3463f6e87b0e9bae4a49934611e43d82d13"}
Jul 29 10:55:52.408130 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:52.407810    2584 scope.go:117] "RemoveContainer" containerID="c9081b2ee40e529e00707e3ed0aa648ff47d48b4540b97779e271e6785a1e4a7"
Jul 29 10:55:52.415927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:52.415909    2584 scope.go:117] "RemoveContainer" containerID="c9081b2ee40e529e00707e3ed0aa648ff47d48b4540b97779e271e6785a1e4a7"
Jul 29 10:55:52.416201 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:55:52.416183    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9081b2ee40e529e00707e3ed0aa648ff47d48b4540b97779e271e6785a1e4a7\": container with ID starting with c9081b2ee40e529e00707e3ed0aa648ff47d48b4540b97779e271e6785a1e4a7 not found: ID does not exist" containerID="c9081b2ee40e529e00707e3ed0aa648ff47d48b4540b97779e271e6785a1e4a7"
Jul 29 10:55:52.416255 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:52.416213    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9081b2ee40e529e00707e3ed0aa648ff47d48b4540b97779e271e6785a1e4a7"} err="failed to get container status \"c9081b2ee40e529e00707e3ed0aa648ff47d48b4540b97779e271e6785a1e4a7\": rpc error: code = NotFound desc = could not find container \"c9081b2ee40e529e00707e3ed0aa648ff47d48b4540b97779e271e6785a1e4a7\": container with ID starting with c9081b2ee40e529e00707e3ed0aa648ff47d48b4540b97779e271e6785a1e4a7 not found: ID does not exist"
Jul 29 10:55:52.424917 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:52.424870    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"]
Jul 29 10:55:52.427258 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:52.427237    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9rhdjb"]
Jul 29 10:55:54.254594 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:55:54.254548    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="625759cb-ae3d-45df-88d4-5948763e6a74" path="/var/lib/kubelet/pods/625759cb-ae3d-45df-88d4-5948763e6a74/volumes"
Jul 29 10:56:08.207624 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.207594    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"]
Jul 29 10:56:08.208262 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.208102    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="625759cb-ae3d-45df-88d4-5948763e6a74" containerName="main"
Jul 29 10:56:08.208262 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.208121    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="625759cb-ae3d-45df-88d4-5948763e6a74" containerName="main"
Jul 29 10:56:08.208262 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.208210    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="625759cb-ae3d-45df-88d4-5948763e6a74" containerName="main"
Jul 29 10:56:08.211332 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.211312    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"
Jul 29 10:56:08.215600 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.215579    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 29 10:56:08.215753 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.215632    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-j98fb\""
Jul 29 10:56:08.215753 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.215660    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 29 10:56:08.215753 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.215633    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 29 10:56:08.215753 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.215586    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-2kl8q\""
Jul 29 10:56:08.220396 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.220376    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"]
Jul 29 10:56:08.242954 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.242922    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc2ee1c-1a67-46f9-85c4-54913ac68a42-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched67jrc\" (UID: \"cdc2ee1c-1a67-46f9-85c4-54913ac68a42\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"
Jul 29 10:56:08.243108 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.243006    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b6nh\" (UniqueName: \"kubernetes.io/projected/cdc2ee1c-1a67-46f9-85c4-54913ac68a42-kube-api-access-7b6nh\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched67jrc\" (UID: \"cdc2ee1c-1a67-46f9-85c4-54913ac68a42\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"
Jul 29 10:56:08.343750 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.343711    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b6nh\" (UniqueName: \"kubernetes.io/projected/cdc2ee1c-1a67-46f9-85c4-54913ac68a42-kube-api-access-7b6nh\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched67jrc\" (UID: \"cdc2ee1c-1a67-46f9-85c4-54913ac68a42\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"
Jul 29 10:56:08.343938 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.343794    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc2ee1c-1a67-46f9-85c4-54913ac68a42-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched67jrc\" (UID: \"cdc2ee1c-1a67-46f9-85c4-54913ac68a42\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"
Jul 29 10:56:08.346272 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.346252    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc2ee1c-1a67-46f9-85c4-54913ac68a42-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched67jrc\" (UID: \"cdc2ee1c-1a67-46f9-85c4-54913ac68a42\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"
Jul 29 10:56:08.352292 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.352266    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b6nh\" (UniqueName: \"kubernetes.io/projected/cdc2ee1c-1a67-46f9-85c4-54913ac68a42-kube-api-access-7b6nh\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched67jrc\" (UID: \"cdc2ee1c-1a67-46f9-85c4-54913ac68a42\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"
Jul 29 10:56:08.521905 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.521816    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"
Jul 29 10:56:08.653291 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:08.653267    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"]
Jul 29 10:56:08.655707 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:56:08.655663    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc2ee1c_1a67_46f9_85c4_54913ac68a42.slice/crio-97c3beef5553a78220181ddebd91022821ed282215a9a4d80309726414795aa1 WatchSource:0}: Error finding container 97c3beef5553a78220181ddebd91022821ed282215a9a4d80309726414795aa1: Status 404 returned error can't find the container with id 97c3beef5553a78220181ddebd91022821ed282215a9a4d80309726414795aa1
Jul 29 10:56:09.485400 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:09.485369    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc" event={"ID":"cdc2ee1c-1a67-46f9-85c4-54913ac68a42","Type":"ContainerStarted","Data":"7a1a20573b2175f3c5d7e470656ff03fda37650ad920f4868516d7044cc19f12"}
Jul 29 10:56:09.485841 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:09.485415    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc" event={"ID":"cdc2ee1c-1a67-46f9-85c4-54913ac68a42","Type":"ContainerStarted","Data":"97c3beef5553a78220181ddebd91022821ed282215a9a4d80309726414795aa1"}
Jul 29 10:56:09.485841 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:09.485599    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"
Jul 29 10:56:09.502607 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:09.502559    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc" podStartSLOduration=1.502538993 podStartE2EDuration="1.502538993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:56:09.500416277 +0000 UTC m=+1637.693443638" watchObservedRunningTime="2026-07-29 10:56:09.502538993 +0000 UTC m=+1637.695566339"
Jul 29 10:56:40.492263 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:40.492233    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"
Jul 29 10:56:42.001343 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:42.001310    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"]
Jul 29 10:56:50.491410 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:50.491375    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc" podUID="cdc2ee1c-1a67-46f9-85c4-54913ac68a42" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:56:57.020833 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.020764    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc" podUID="cdc2ee1c-1a67-46f9-85c4-54913ac68a42" containerName="main" containerID="cri-o://7a1a20573b2175f3c5d7e470656ff03fda37650ad920f4868516d7044cc19f12" gracePeriod=45
Jul 29 10:56:57.277270 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.277210    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"
Jul 29 10:56:57.380487 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.380445    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7b6nh\" (UniqueName: \"kubernetes.io/projected/cdc2ee1c-1a67-46f9-85c4-54913ac68a42-kube-api-access-7b6nh\") pod \"cdc2ee1c-1a67-46f9-85c4-54913ac68a42\" (UID: \"cdc2ee1c-1a67-46f9-85c4-54913ac68a42\") "
Jul 29 10:56:57.380702 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.380560    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc2ee1c-1a67-46f9-85c4-54913ac68a42-tls-certs\") pod \"cdc2ee1c-1a67-46f9-85c4-54913ac68a42\" (UID: \"cdc2ee1c-1a67-46f9-85c4-54913ac68a42\") "
Jul 29 10:56:57.382571 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.382542    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc2ee1c-1a67-46f9-85c4-54913ac68a42-kube-api-access-7b6nh" (OuterVolumeSpecName: "kube-api-access-7b6nh") pod "cdc2ee1c-1a67-46f9-85c4-54913ac68a42" (UID: "cdc2ee1c-1a67-46f9-85c4-54913ac68a42"). InnerVolumeSpecName "kube-api-access-7b6nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:56:57.382670 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.382618    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc2ee1c-1a67-46f9-85c4-54913ac68a42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cdc2ee1c-1a67-46f9-85c4-54913ac68a42" (UID: "cdc2ee1c-1a67-46f9-85c4-54913ac68a42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:56:57.482177 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.482141    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b6nh\" (UniqueName: \"kubernetes.io/projected/cdc2ee1c-1a67-46f9-85c4-54913ac68a42-kube-api-access-7b6nh\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:56:57.482177 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.482172    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc2ee1c-1a67-46f9-85c4-54913ac68a42-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:56:57.707656 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.707624    2584 generic.go:356] "Generic (PLEG): container finished" podID="cdc2ee1c-1a67-46f9-85c4-54913ac68a42" containerID="7a1a20573b2175f3c5d7e470656ff03fda37650ad920f4868516d7044cc19f12" exitCode=0
Jul 29 10:56:57.707879 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.707670    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"
Jul 29 10:56:57.707879 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.707715    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc" event={"ID":"cdc2ee1c-1a67-46f9-85c4-54913ac68a42","Type":"ContainerDied","Data":"7a1a20573b2175f3c5d7e470656ff03fda37650ad920f4868516d7044cc19f12"}
Jul 29 10:56:57.707879 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.707803    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc" event={"ID":"cdc2ee1c-1a67-46f9-85c4-54913ac68a42","Type":"ContainerDied","Data":"97c3beef5553a78220181ddebd91022821ed282215a9a4d80309726414795aa1"}
Jul 29 10:56:57.707879 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.707834    2584 scope.go:117] "RemoveContainer" containerID="7a1a20573b2175f3c5d7e470656ff03fda37650ad920f4868516d7044cc19f12"
Jul 29 10:56:57.716541 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.716518    2584 scope.go:117] "RemoveContainer" containerID="7a1a20573b2175f3c5d7e470656ff03fda37650ad920f4868516d7044cc19f12"
Jul 29 10:56:57.716816 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:56:57.716797    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1a20573b2175f3c5d7e470656ff03fda37650ad920f4868516d7044cc19f12\": container with ID starting with 7a1a20573b2175f3c5d7e470656ff03fda37650ad920f4868516d7044cc19f12 not found: ID does not exist" containerID="7a1a20573b2175f3c5d7e470656ff03fda37650ad920f4868516d7044cc19f12"
Jul 29 10:56:57.716867 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.716825    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1a20573b2175f3c5d7e470656ff03fda37650ad920f4868516d7044cc19f12"} err="failed to get container status \"7a1a20573b2175f3c5d7e470656ff03fda37650ad920f4868516d7044cc19f12\": rpc error: code = NotFound desc = could not find container \"7a1a20573b2175f3c5d7e470656ff03fda37650ad920f4868516d7044cc19f12\": container with ID starting with 7a1a20573b2175f3c5d7e470656ff03fda37650ad920f4868516d7044cc19f12 not found: ID does not exist"
Jul 29 10:56:57.729116 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.729095    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"]
Jul 29 10:56:57.732459 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:57.732439    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched67jrc"]
Jul 29 10:56:58.255186 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:56:58.255148    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdc2ee1c-1a67-46f9-85c4-54913ac68a42" path="/var/lib/kubelet/pods/cdc2ee1c-1a67-46f9-85c4-54913ac68a42/volumes"
Jul 29 10:57:08.042501 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.042470    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"]
Jul 29 10:57:08.042939 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.042832    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdc2ee1c-1a67-46f9-85c4-54913ac68a42" containerName="main"
Jul 29 10:57:08.042939 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.042844    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdc2ee1c-1a67-46f9-85c4-54913ac68a42" containerName="main"
Jul 29 10:57:08.042939 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.042916    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdc2ee1c-1a67-46f9-85c4-54913ac68a42" containerName="main"
Jul 29 10:57:08.047343 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.047322    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.050544 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.050513    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 29 10:57:08.051040 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.051019    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-gj6v8\""
Jul 29 10:57:08.051170 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.051064    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 29 10:57:08.052377 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.052358    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 29 10:57:08.052489 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.052476    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 29 10:57:08.059226 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.059206    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"]
Jul 29 10:57:08.180697 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.180650    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7f98a8-2838-4457-9acd-006a873b9711-tls-certs\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.180900 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.180721    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-dshm\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.180900 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.180815    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.180900 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.180891    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-home\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.181084 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.180915    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.181084 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.181048    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-model-cache\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.181084 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.181076    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6vbv\" (UniqueName: \"kubernetes.io/projected/8e7f98a8-2838-4457-9acd-006a873b9711-kube-api-access-d6vbv\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.281922 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.281895    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-model-cache\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.282106 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.281928    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6vbv\" (UniqueName: \"kubernetes.io/projected/8e7f98a8-2838-4457-9acd-006a873b9711-kube-api-access-d6vbv\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.282106 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.281966    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7f98a8-2838-4457-9acd-006a873b9711-tls-certs\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.282106 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.281996    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-dshm\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.282106 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.282030    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.282106 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.282066    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-home\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.282106 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.282087    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.282429 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.282324    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-model-cache\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.282429 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.282366    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.282510 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.282427    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-home\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.282510 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.282443    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.284332 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.284311    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-dshm\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.284517 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.284498    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7f98a8-2838-4457-9acd-006a873b9711-tls-certs\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.290288 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.290258    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6vbv\" (UniqueName: \"kubernetes.io/projected/8e7f98a8-2838-4457-9acd-006a873b9711-kube-api-access-d6vbv\") pod \"tokenizer-clean-path-test-kserve-768597f86d-hp7z4\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.340387 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.340312    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"]
Jul 29 10:57:08.343798 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.343780    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"
Jul 29 10:57:08.354806 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.354784    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"]
Jul 29 10:57:08.358444 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.358423    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:08.484311 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.484281    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55d136f-7ead-4c42-959d-4fbb0dcab187-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj\" (UID: \"c55d136f-7ead-4c42-959d-4fbb0dcab187\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"
Jul 29 10:57:08.484474 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.484336    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55d136f-7ead-4c42-959d-4fbb0dcab187-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj\" (UID: \"c55d136f-7ead-4c42-959d-4fbb0dcab187\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"
Jul 29 10:57:08.484474 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.484361    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55d136f-7ead-4c42-959d-4fbb0dcab187-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj\" (UID: \"c55d136f-7ead-4c42-959d-4fbb0dcab187\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"
Jul 29 10:57:08.497743 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.497720    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"]
Jul 29 10:57:08.500022 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:57:08.499996    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7f98a8_2838_4457_9acd_006a873b9711.slice/crio-fac7d94b4bf9d5487f7952e5063b654707527b923241cce7bc6ed630a4fdd93a WatchSource:0}: Error finding container fac7d94b4bf9d5487f7952e5063b654707527b923241cce7bc6ed630a4fdd93a: Status 404 returned error can't find the container with id fac7d94b4bf9d5487f7952e5063b654707527b923241cce7bc6ed630a4fdd93a
Jul 29 10:57:08.585729 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.585696    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55d136f-7ead-4c42-959d-4fbb0dcab187-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj\" (UID: \"c55d136f-7ead-4c42-959d-4fbb0dcab187\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"
Jul 29 10:57:08.585878 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.585759    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55d136f-7ead-4c42-959d-4fbb0dcab187-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj\" (UID: \"c55d136f-7ead-4c42-959d-4fbb0dcab187\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"
Jul 29 10:57:08.585964 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.585945    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55d136f-7ead-4c42-959d-4fbb0dcab187-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj\" (UID: \"c55d136f-7ead-4c42-959d-4fbb0dcab187\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"
Jul 29 10:57:08.586171 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.586144    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55d136f-7ead-4c42-959d-4fbb0dcab187-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj\" (UID: \"c55d136f-7ead-4c42-959d-4fbb0dcab187\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"
Jul 29 10:57:08.586398 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.586376    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55d136f-7ead-4c42-959d-4fbb0dcab187-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj\" (UID: \"c55d136f-7ead-4c42-959d-4fbb0dcab187\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"
Jul 29 10:57:08.588398 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.588369    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55d136f-7ead-4c42-959d-4fbb0dcab187-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj\" (UID: \"c55d136f-7ead-4c42-959d-4fbb0dcab187\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"
Jul 29 10:57:08.654121 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.654038    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"
Jul 29 10:57:08.775982 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.775944    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4" event={"ID":"8e7f98a8-2838-4457-9acd-006a873b9711","Type":"ContainerStarted","Data":"fc88290832669ce1dd16d1561f4c6b9a311bb7761f9c82b9d50b32e22545d9d4"}
Jul 29 10:57:08.776130 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.776039    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4" event={"ID":"8e7f98a8-2838-4457-9acd-006a873b9711","Type":"ContainerStarted","Data":"fac7d94b4bf9d5487f7952e5063b654707527b923241cce7bc6ed630a4fdd93a"}
Jul 29 10:57:08.793690 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:08.793659    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"]
Jul 29 10:57:08.796396 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:57:08.796371    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55d136f_7ead_4c42_959d_4fbb0dcab187.slice/crio-8de7e0d26836740d84ae6f5b5ecd876528c6fb16b986b6fc9afcad075234c1aa WatchSource:0}: Error finding container 8de7e0d26836740d84ae6f5b5ecd876528c6fb16b986b6fc9afcad075234c1aa: Status 404 returned error can't find the container with id 8de7e0d26836740d84ae6f5b5ecd876528c6fb16b986b6fc9afcad075234c1aa
Jul 29 10:57:09.785340 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:09.785296    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj" event={"ID":"c55d136f-7ead-4c42-959d-4fbb0dcab187","Type":"ContainerStarted","Data":"e04e4b709e39955b302dd117555b50975a22c1d5f28b45f3d080c685f4ad4e2d"}
Jul 29 10:57:09.785927 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:09.785351    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj" event={"ID":"c55d136f-7ead-4c42-959d-4fbb0dcab187","Type":"ContainerStarted","Data":"8de7e0d26836740d84ae6f5b5ecd876528c6fb16b986b6fc9afcad075234c1aa"}
Jul 29 10:57:12.803746 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:12.803708    2584 generic.go:356] "Generic (PLEG): container finished" podID="c55d136f-7ead-4c42-959d-4fbb0dcab187" containerID="e04e4b709e39955b302dd117555b50975a22c1d5f28b45f3d080c685f4ad4e2d" exitCode=0
Jul 29 10:57:12.804229 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:12.803769    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj" event={"ID":"c55d136f-7ead-4c42-959d-4fbb0dcab187","Type":"ContainerDied","Data":"e04e4b709e39955b302dd117555b50975a22c1d5f28b45f3d080c685f4ad4e2d"}
Jul 29 10:57:12.805216 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:12.805195    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:57:13.812069 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:13.811811    2584 generic.go:356] "Generic (PLEG): container finished" podID="8e7f98a8-2838-4457-9acd-006a873b9711" containerID="fc88290832669ce1dd16d1561f4c6b9a311bb7761f9c82b9d50b32e22545d9d4" exitCode=0
Jul 29 10:57:13.812069 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:13.811934    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4" event={"ID":"8e7f98a8-2838-4457-9acd-006a873b9711","Type":"ContainerDied","Data":"fc88290832669ce1dd16d1561f4c6b9a311bb7761f9c82b9d50b32e22545d9d4"}
Jul 29 10:57:14.820691 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:14.820635    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4" event={"ID":"8e7f98a8-2838-4457-9acd-006a873b9711","Type":"ContainerStarted","Data":"968bb8d42d38ba255c89924fe1c179f4d9c587a77f4c698f1c5fd5228e5c3211"}
Jul 29 10:57:14.841993 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:14.841935    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4" podStartSLOduration=2.5115924720000002 podStartE2EDuration="6.841915076s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.330322604s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:57:14.839671917 +0000 UTC m=+1703.032699265" watchObservedRunningTime="2026-07-29 10:57:14.841915076 +0000 UTC m=+1703.034942424"
Jul 29 10:57:15.359481 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:15.359453    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:18.358906 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:18.358869    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:18.378817 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:18.378783    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:18.857672 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:18.857645    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:57:46.995332 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:46.995295    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj" event={"ID":"c55d136f-7ead-4c42-959d-4fbb0dcab187","Type":"ContainerStarted","Data":"49a205793684329e8a7b6f8a70d1d253b1142853aedb97d6d7762fe887af3ca4"}
Jul 29 10:57:46.995756 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:46.995528    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"
Jul 29 10:57:47.015338 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:57:47.015281    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj" podStartSLOduration=1.779432568 podStartE2EDuration="39.015260257s" totalImagesPullingTime="33.375194827s" totalInitContainerRuntime="3.860632862s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:57:47.013385985 +0000 UTC m=+1735.206413335" watchObservedRunningTime="2026-07-29 10:57:47.015260257 +0000 UTC m=+1735.208287605"
Jul 29 10:58:18.013618 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:18.013590    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"
Jul 29 10:58:18.977891 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:18.977865    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj_c55d136f-7ead-4c42-959d-4fbb0dcab187/main/0.log"
Jul 29 10:58:19.478869 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.478835    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"]
Jul 29 10:58:19.485999 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.485970    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"]
Jul 29 10:58:19.486218 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.486196    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj" podUID="c55d136f-7ead-4c42-959d-4fbb0dcab187" containerName="main" containerID="cri-o://49a205793684329e8a7b6f8a70d1d253b1142853aedb97d6d7762fe887af3ca4" gracePeriod=30
Jul 29 10:58:19.488862 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:58:19.488824    2584 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:58:19.488862 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:58:19.488862 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:58:19.488862 ip-10-0-140-240 kubenswrapper[2584]:  > containerID="968bb8d42d38ba255c89924fe1c179f4d9c587a77f4c698f1c5fd5228e5c3211" cmd=["/bin/sleep","15"]
Jul 29 10:58:19.489135 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.488878    2584 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4" message="" err=<
Jul 29 10:58:19.489135 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:58:19.489135 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:58:19.489135 ip-10-0-140-240 kubenswrapper[2584]:  >
Jul 29 10:58:19.489135 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:58:19.488893    2584 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:58:19.489135 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:58:19.489135 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:58:19.489135 ip-10-0-140-240 kubenswrapper[2584]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4" podUID="8e7f98a8-2838-4457-9acd-006a873b9711" containerName="main" containerID="cri-o://968bb8d42d38ba255c89924fe1c179f4d9c587a77f4c698f1c5fd5228e5c3211"
Jul 29 10:58:19.489135 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.488934    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4" podUID="8e7f98a8-2838-4457-9acd-006a873b9711" containerName="main" containerID="cri-o://968bb8d42d38ba255c89924fe1c179f4d9c587a77f4c698f1c5fd5228e5c3211" gracePeriod=60
Jul 29 10:58:19.858186 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.858158    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:58:19.951656 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.951630    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-tmp-dir\") pod \"8e7f98a8-2838-4457-9acd-006a873b9711\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") "
Jul 29 10:58:19.951785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.951706    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-home\") pod \"8e7f98a8-2838-4457-9acd-006a873b9711\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") "
Jul 29 10:58:19.951785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.951742    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6vbv\" (UniqueName: \"kubernetes.io/projected/8e7f98a8-2838-4457-9acd-006a873b9711-kube-api-access-d6vbv\") pod \"8e7f98a8-2838-4457-9acd-006a873b9711\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") "
Jul 29 10:58:19.951871 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.951787    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-model-cache\") pod \"8e7f98a8-2838-4457-9acd-006a873b9711\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") "
Jul 29 10:58:19.951871 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.951824    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7f98a8-2838-4457-9acd-006a873b9711-tls-certs\") pod \"8e7f98a8-2838-4457-9acd-006a873b9711\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") "
Jul 29 10:58:19.951961 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.951909    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-kserve-provision-location\") pod \"8e7f98a8-2838-4457-9acd-006a873b9711\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") "
Jul 29 10:58:19.951961 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.951907    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e7f98a8-2838-4457-9acd-006a873b9711" (UID: "8e7f98a8-2838-4457-9acd-006a873b9711"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:58:19.951961 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.951944    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-dshm\") pod \"8e7f98a8-2838-4457-9acd-006a873b9711\" (UID: \"8e7f98a8-2838-4457-9acd-006a873b9711\") "
Jul 29 10:58:19.952115 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.951983    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-home" (OuterVolumeSpecName: "home") pod "8e7f98a8-2838-4457-9acd-006a873b9711" (UID: "8e7f98a8-2838-4457-9acd-006a873b9711"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:58:19.952115 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.951995    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e7f98a8-2838-4457-9acd-006a873b9711" (UID: "8e7f98a8-2838-4457-9acd-006a873b9711"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:58:19.952225 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.952207    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:58:19.952281 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.952225    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-home\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:58:19.952281 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.952238    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-model-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:58:19.954067 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.954036    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7f98a8-2838-4457-9acd-006a873b9711-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e7f98a8-2838-4457-9acd-006a873b9711" (UID: "8e7f98a8-2838-4457-9acd-006a873b9711"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:58:19.954214 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.954193    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7f98a8-2838-4457-9acd-006a873b9711-kube-api-access-d6vbv" (OuterVolumeSpecName: "kube-api-access-d6vbv") pod "8e7f98a8-2838-4457-9acd-006a873b9711" (UID: "8e7f98a8-2838-4457-9acd-006a873b9711"). InnerVolumeSpecName "kube-api-access-d6vbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:58:19.954498 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:19.954476    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-dshm" (OuterVolumeSpecName: "dshm") pod "8e7f98a8-2838-4457-9acd-006a873b9711" (UID: "8e7f98a8-2838-4457-9acd-006a873b9711"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:58:20.011529 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.011496    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e7f98a8-2838-4457-9acd-006a873b9711" (UID: "8e7f98a8-2838-4457-9acd-006a873b9711"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:58:20.053464 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.053437    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6vbv\" (UniqueName: \"kubernetes.io/projected/8e7f98a8-2838-4457-9acd-006a873b9711-kube-api-access-d6vbv\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:58:20.053464 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.053467    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7f98a8-2838-4457-9acd-006a873b9711-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:58:20.053660 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.053479    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-kserve-provision-location\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:58:20.053660 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.053488    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7f98a8-2838-4457-9acd-006a873b9711-dshm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:58:20.135646 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.135622    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"
Jul 29 10:58:20.163818 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.163790    2584 generic.go:356] "Generic (PLEG): container finished" podID="8e7f98a8-2838-4457-9acd-006a873b9711" containerID="968bb8d42d38ba255c89924fe1c179f4d9c587a77f4c698f1c5fd5228e5c3211" exitCode=0
Jul 29 10:58:20.163983 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.163837    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"
Jul 29 10:58:20.163983 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.163870    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4" event={"ID":"8e7f98a8-2838-4457-9acd-006a873b9711","Type":"ContainerDied","Data":"968bb8d42d38ba255c89924fe1c179f4d9c587a77f4c698f1c5fd5228e5c3211"}
Jul 29 10:58:20.163983 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.163927    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4" event={"ID":"8e7f98a8-2838-4457-9acd-006a873b9711","Type":"ContainerDied","Data":"fac7d94b4bf9d5487f7952e5063b654707527b923241cce7bc6ed630a4fdd93a"}
Jul 29 10:58:20.163983 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.163959    2584 scope.go:117] "RemoveContainer" containerID="968bb8d42d38ba255c89924fe1c179f4d9c587a77f4c698f1c5fd5228e5c3211"
Jul 29 10:58:20.165660 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.165636    2584 generic.go:356] "Generic (PLEG): container finished" podID="c55d136f-7ead-4c42-959d-4fbb0dcab187" containerID="49a205793684329e8a7b6f8a70d1d253b1142853aedb97d6d7762fe887af3ca4" exitCode=0
Jul 29 10:58:20.165796 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.165707    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj" event={"ID":"c55d136f-7ead-4c42-959d-4fbb0dcab187","Type":"ContainerDied","Data":"49a205793684329e8a7b6f8a70d1d253b1142853aedb97d6d7762fe887af3ca4"}
Jul 29 10:58:20.165796 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.165726    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"
Jul 29 10:58:20.165796 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.165760    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj" event={"ID":"c55d136f-7ead-4c42-959d-4fbb0dcab187","Type":"ContainerDied","Data":"8de7e0d26836740d84ae6f5b5ecd876528c6fb16b986b6fc9afcad075234c1aa"}
Jul 29 10:58:20.174669 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.174647    2584 scope.go:117] "RemoveContainer" containerID="fc88290832669ce1dd16d1561f4c6b9a311bb7761f9c82b9d50b32e22545d9d4"
Jul 29 10:58:20.182725 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.182710    2584 scope.go:117] "RemoveContainer" containerID="968bb8d42d38ba255c89924fe1c179f4d9c587a77f4c698f1c5fd5228e5c3211"
Jul 29 10:58:20.182980 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:58:20.182954    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968bb8d42d38ba255c89924fe1c179f4d9c587a77f4c698f1c5fd5228e5c3211\": container with ID starting with 968bb8d42d38ba255c89924fe1c179f4d9c587a77f4c698f1c5fd5228e5c3211 not found: ID does not exist" containerID="968bb8d42d38ba255c89924fe1c179f4d9c587a77f4c698f1c5fd5228e5c3211"
Jul 29 10:58:20.183058 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.182982    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968bb8d42d38ba255c89924fe1c179f4d9c587a77f4c698f1c5fd5228e5c3211"} err="failed to get container status \"968bb8d42d38ba255c89924fe1c179f4d9c587a77f4c698f1c5fd5228e5c3211\": rpc error: code = NotFound desc = could not find container \"968bb8d42d38ba255c89924fe1c179f4d9c587a77f4c698f1c5fd5228e5c3211\": container with ID starting with 968bb8d42d38ba255c89924fe1c179f4d9c587a77f4c698f1c5fd5228e5c3211 not found: ID does not exist"
Jul 29 10:58:20.183058 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.183003    2584 scope.go:117] "RemoveContainer" containerID="fc88290832669ce1dd16d1561f4c6b9a311bb7761f9c82b9d50b32e22545d9d4"
Jul 29 10:58:20.183256 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:58:20.183227    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc88290832669ce1dd16d1561f4c6b9a311bb7761f9c82b9d50b32e22545d9d4\": container with ID starting with fc88290832669ce1dd16d1561f4c6b9a311bb7761f9c82b9d50b32e22545d9d4 not found: ID does not exist" containerID="fc88290832669ce1dd16d1561f4c6b9a311bb7761f9c82b9d50b32e22545d9d4"
Jul 29 10:58:20.183323 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.183256    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc88290832669ce1dd16d1561f4c6b9a311bb7761f9c82b9d50b32e22545d9d4"} err="failed to get container status \"fc88290832669ce1dd16d1561f4c6b9a311bb7761f9c82b9d50b32e22545d9d4\": rpc error: code = NotFound desc = could not find container \"fc88290832669ce1dd16d1561f4c6b9a311bb7761f9c82b9d50b32e22545d9d4\": container with ID starting with fc88290832669ce1dd16d1561f4c6b9a311bb7761f9c82b9d50b32e22545d9d4 not found: ID does not exist"
Jul 29 10:58:20.183323 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.183281    2584 scope.go:117] "RemoveContainer" containerID="49a205793684329e8a7b6f8a70d1d253b1142853aedb97d6d7762fe887af3ca4"
Jul 29 10:58:20.185302 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.185282    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"]
Jul 29 10:58:20.188970 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.188951    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4"]
Jul 29 10:58:20.192381 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.192365    2584 scope.go:117] "RemoveContainer" containerID="e04e4b709e39955b302dd117555b50975a22c1d5f28b45f3d080c685f4ad4e2d"
Jul 29 10:58:20.199594 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.199573    2584 scope.go:117] "RemoveContainer" containerID="49a205793684329e8a7b6f8a70d1d253b1142853aedb97d6d7762fe887af3ca4"
Jul 29 10:58:20.199844 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:58:20.199824    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a205793684329e8a7b6f8a70d1d253b1142853aedb97d6d7762fe887af3ca4\": container with ID starting with 49a205793684329e8a7b6f8a70d1d253b1142853aedb97d6d7762fe887af3ca4 not found: ID does not exist" containerID="49a205793684329e8a7b6f8a70d1d253b1142853aedb97d6d7762fe887af3ca4"
Jul 29 10:58:20.199921 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.199852    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a205793684329e8a7b6f8a70d1d253b1142853aedb97d6d7762fe887af3ca4"} err="failed to get container status \"49a205793684329e8a7b6f8a70d1d253b1142853aedb97d6d7762fe887af3ca4\": rpc error: code = NotFound desc = could not find container \"49a205793684329e8a7b6f8a70d1d253b1142853aedb97d6d7762fe887af3ca4\": container with ID starting with 49a205793684329e8a7b6f8a70d1d253b1142853aedb97d6d7762fe887af3ca4 not found: ID does not exist"
Jul 29 10:58:20.199921 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.199874    2584 scope.go:117] "RemoveContainer" containerID="e04e4b709e39955b302dd117555b50975a22c1d5f28b45f3d080c685f4ad4e2d"
Jul 29 10:58:20.200116 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:58:20.200100    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04e4b709e39955b302dd117555b50975a22c1d5f28b45f3d080c685f4ad4e2d\": container with ID starting with e04e4b709e39955b302dd117555b50975a22c1d5f28b45f3d080c685f4ad4e2d not found: ID does not exist" containerID="e04e4b709e39955b302dd117555b50975a22c1d5f28b45f3d080c685f4ad4e2d"
Jul 29 10:58:20.200156 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.200121    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04e4b709e39955b302dd117555b50975a22c1d5f28b45f3d080c685f4ad4e2d"} err="failed to get container status \"e04e4b709e39955b302dd117555b50975a22c1d5f28b45f3d080c685f4ad4e2d\": rpc error: code = NotFound desc = could not find container \"e04e4b709e39955b302dd117555b50975a22c1d5f28b45f3d080c685f4ad4e2d\": container with ID starting with e04e4b709e39955b302dd117555b50975a22c1d5f28b45f3d080c685f4ad4e2d not found: ID does not exist"
Jul 29 10:58:20.254641 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.254608    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e7f98a8-2838-4457-9acd-006a873b9711" path="/var/lib/kubelet/pods/8e7f98a8-2838-4457-9acd-006a873b9711/volumes"
Jul 29 10:58:20.255230 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.255204    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55d136f-7ead-4c42-959d-4fbb0dcab187-tls-certs\") pod \"c55d136f-7ead-4c42-959d-4fbb0dcab187\" (UID: \"c55d136f-7ead-4c42-959d-4fbb0dcab187\") "
Jul 29 10:58:20.255230 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.255233    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55d136f-7ead-4c42-959d-4fbb0dcab187-tmp-dir\") pod \"c55d136f-7ead-4c42-959d-4fbb0dcab187\" (UID: \"c55d136f-7ead-4c42-959d-4fbb0dcab187\") "
Jul 29 10:58:20.255407 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.255253    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55d136f-7ead-4c42-959d-4fbb0dcab187-kserve-provision-location\") pod \"c55d136f-7ead-4c42-959d-4fbb0dcab187\" (UID: \"c55d136f-7ead-4c42-959d-4fbb0dcab187\") "
Jul 29 10:58:20.255622 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.255601    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55d136f-7ead-4c42-959d-4fbb0dcab187-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c55d136f-7ead-4c42-959d-4fbb0dcab187" (UID: "c55d136f-7ead-4c42-959d-4fbb0dcab187"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:58:20.257372 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.257351    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55d136f-7ead-4c42-959d-4fbb0dcab187-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c55d136f-7ead-4c42-959d-4fbb0dcab187" (UID: "c55d136f-7ead-4c42-959d-4fbb0dcab187"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:58:20.313134 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.313033    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55d136f-7ead-4c42-959d-4fbb0dcab187-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c55d136f-7ead-4c42-959d-4fbb0dcab187" (UID: "c55d136f-7ead-4c42-959d-4fbb0dcab187"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:58:20.356696 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.356651    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55d136f-7ead-4c42-959d-4fbb0dcab187-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:58:20.356696 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.356696    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55d136f-7ead-4c42-959d-4fbb0dcab187-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:58:20.356887 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.356707    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55d136f-7ead-4c42-959d-4fbb0dcab187-kserve-provision-location\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:58:20.488123 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.488093    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"]
Jul 29 10:58:20.491789 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.491768    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67747db44f-hl2fj"]
Jul 29 10:58:20.844561 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:20.844521    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-hp7z4" podUID="8e7f98a8-2838-4457-9acd-006a873b9711" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 10:58:22.254598 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:22.254566    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c55d136f-7ead-4c42-959d-4fbb0dcab187" path="/var/lib/kubelet/pods/c55d136f-7ead-4c42-959d-4fbb0dcab187/volumes"
Jul 29 10:58:46.049443 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.049363    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"]
Jul 29 10:58:46.049785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.049712    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7f98a8-2838-4457-9acd-006a873b9711" containerName="main"
Jul 29 10:58:46.049785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.049724    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7f98a8-2838-4457-9acd-006a873b9711" containerName="main"
Jul 29 10:58:46.049785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.049737    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7f98a8-2838-4457-9acd-006a873b9711" containerName="storage-initializer"
Jul 29 10:58:46.049785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.049743    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7f98a8-2838-4457-9acd-006a873b9711" containerName="storage-initializer"
Jul 29 10:58:46.049785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.049752    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c55d136f-7ead-4c42-959d-4fbb0dcab187" containerName="storage-initializer"
Jul 29 10:58:46.049785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.049759    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c55d136f-7ead-4c42-959d-4fbb0dcab187" containerName="storage-initializer"
Jul 29 10:58:46.049785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.049770    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c55d136f-7ead-4c42-959d-4fbb0dcab187" containerName="main"
Jul 29 10:58:46.049785 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.049776    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c55d136f-7ead-4c42-959d-4fbb0dcab187" containerName="main"
Jul 29 10:58:46.050081 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.049834    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c55d136f-7ead-4c42-959d-4fbb0dcab187" containerName="main"
Jul 29 10:58:46.050081 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.049848    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7f98a8-2838-4457-9acd-006a873b9711" containerName="main"
Jul 29 10:58:46.053109 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.053088    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.056479 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.056458    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 29 10:58:46.057764 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.057737    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-wftkd\""
Jul 29 10:58:46.057764 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.057757    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 29 10:58:46.057981 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.057807    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 29 10:58:46.057981 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.057864    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 29 10:58:46.073273 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.073248    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"]
Jul 29 10:58:46.179031 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.178999    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/321d53a5-474d-4842-8831-abe7afc161e3-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.179233 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.179043    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vvhw\" (UniqueName: \"kubernetes.io/projected/321d53a5-474d-4842-8831-abe7afc161e3-kube-api-access-2vvhw\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.179233 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.179065    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-home\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.179233 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.179179    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.179413 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.179239    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.179413 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.179281    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-model-cache\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.179413 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.179347    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-dshm\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.280717 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.280670    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/321d53a5-474d-4842-8831-abe7afc161e3-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.280889 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.280742    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vvhw\" (UniqueName: \"kubernetes.io/projected/321d53a5-474d-4842-8831-abe7afc161e3-kube-api-access-2vvhw\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.280889 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.280772    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-home\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.280889 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.280825    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.280889 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.280875    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.281085 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.280911    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-model-cache\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.281085 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.280959    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-dshm\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.281298 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.281274    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-home\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.281376 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.281295    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.281376 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.281348    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.281376 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.281365    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-model-cache\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.283148 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.283129    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-dshm\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.283396 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.283376    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/321d53a5-474d-4842-8831-abe7afc161e3-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.291621 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.291577    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vvhw\" (UniqueName: \"kubernetes.io/projected/321d53a5-474d-4842-8831-abe7afc161e3-kube-api-access-2vvhw\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.310241 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.310173    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"]
Jul 29 10:58:46.313902 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.313883    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"
Jul 29 10:58:46.328068 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.328045    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"]
Jul 29 10:58:46.362650 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.362623    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:46.482854 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.482821    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6697752f-3304-469e-87b1-05a296014c65-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk\" (UID: \"6697752f-3304-469e-87b1-05a296014c65\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"
Jul 29 10:58:46.483032 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.482872    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6697752f-3304-469e-87b1-05a296014c65-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk\" (UID: \"6697752f-3304-469e-87b1-05a296014c65\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"
Jul 29 10:58:46.483032 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.482999    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6697752f-3304-469e-87b1-05a296014c65-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk\" (UID: \"6697752f-3304-469e-87b1-05a296014c65\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"
Jul 29 10:58:46.497602 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.497580    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"]
Jul 29 10:58:46.500232 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:58:46.500198    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321d53a5_474d_4842_8831_abe7afc161e3.slice/crio-d0d8c1125565a1be8c74181a4b9a57b15c54d424feed17a6f736a65f96bcdc21 WatchSource:0}: Error finding container d0d8c1125565a1be8c74181a4b9a57b15c54d424feed17a6f736a65f96bcdc21: Status 404 returned error can't find the container with id d0d8c1125565a1be8c74181a4b9a57b15c54d424feed17a6f736a65f96bcdc21
Jul 29 10:58:46.584283 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.584196    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6697752f-3304-469e-87b1-05a296014c65-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk\" (UID: \"6697752f-3304-469e-87b1-05a296014c65\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"
Jul 29 10:58:46.584283 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.584255    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6697752f-3304-469e-87b1-05a296014c65-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk\" (UID: \"6697752f-3304-469e-87b1-05a296014c65\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"
Jul 29 10:58:46.584694 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.584325    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6697752f-3304-469e-87b1-05a296014c65-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk\" (UID: \"6697752f-3304-469e-87b1-05a296014c65\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"
Jul 29 10:58:46.584694 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.584641    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6697752f-3304-469e-87b1-05a296014c65-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk\" (UID: \"6697752f-3304-469e-87b1-05a296014c65\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"
Jul 29 10:58:46.584800 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.584710    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6697752f-3304-469e-87b1-05a296014c65-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk\" (UID: \"6697752f-3304-469e-87b1-05a296014c65\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"
Jul 29 10:58:46.587113 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.587088    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6697752f-3304-469e-87b1-05a296014c65-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk\" (UID: \"6697752f-3304-469e-87b1-05a296014c65\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"
Jul 29 10:58:46.623988 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.623956    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"
Jul 29 10:58:46.767833 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:46.767806    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"]
Jul 29 10:58:46.768570 ip-10-0-140-240 kubenswrapper[2584]: W0729 10:58:46.768540    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6697752f_3304_469e_87b1_05a296014c65.slice/crio-7dc04947266859231e8c059b55d8fac8df36241542a936931d1e01df93f29b1b WatchSource:0}: Error finding container 7dc04947266859231e8c059b55d8fac8df36241542a936931d1e01df93f29b1b: Status 404 returned error can't find the container with id 7dc04947266859231e8c059b55d8fac8df36241542a936931d1e01df93f29b1b
Jul 29 10:58:47.295283 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:47.295246    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk" event={"ID":"6697752f-3304-469e-87b1-05a296014c65","Type":"ContainerStarted","Data":"3c5d1020b6fc6a3e3583b6f7f359623c90508155e3da0f9e48aa0c1598ebde24"}
Jul 29 10:58:47.295743 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:47.295312    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk" event={"ID":"6697752f-3304-469e-87b1-05a296014c65","Type":"ContainerStarted","Data":"7dc04947266859231e8c059b55d8fac8df36241542a936931d1e01df93f29b1b"}
Jul 29 10:58:47.299131 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:47.299101    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8" event={"ID":"321d53a5-474d-4842-8831-abe7afc161e3","Type":"ContainerStarted","Data":"ff2bfe366b5d938c8bc2844433a633c75536972490880b737710c432396bc963"}
Jul 29 10:58:47.299568 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:47.299542    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8" event={"ID":"321d53a5-474d-4842-8831-abe7afc161e3","Type":"ContainerStarted","Data":"d0d8c1125565a1be8c74181a4b9a57b15c54d424feed17a6f736a65f96bcdc21"}
Jul 29 10:58:51.327309 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:51.327278    2584 generic.go:356] "Generic (PLEG): container finished" podID="6697752f-3304-469e-87b1-05a296014c65" containerID="3c5d1020b6fc6a3e3583b6f7f359623c90508155e3da0f9e48aa0c1598ebde24" exitCode=0
Jul 29 10:58:51.327827 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:51.327353    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk" event={"ID":"6697752f-3304-469e-87b1-05a296014c65","Type":"ContainerDied","Data":"3c5d1020b6fc6a3e3583b6f7f359623c90508155e3da0f9e48aa0c1598ebde24"}
Jul 29 10:58:51.328965 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:51.328945    2584 generic.go:356] "Generic (PLEG): container finished" podID="321d53a5-474d-4842-8831-abe7afc161e3" containerID="ff2bfe366b5d938c8bc2844433a633c75536972490880b737710c432396bc963" exitCode=0
Jul 29 10:58:51.329065 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:51.328980    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8" event={"ID":"321d53a5-474d-4842-8831-abe7afc161e3","Type":"ContainerDied","Data":"ff2bfe366b5d938c8bc2844433a633c75536972490880b737710c432396bc963"}
Jul 29 10:58:52.334623 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:52.334584    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8" event={"ID":"321d53a5-474d-4842-8831-abe7afc161e3","Type":"ContainerStarted","Data":"0b46df227effc4e0d4dc404454eeec730d0d308b8dfaece74064c97caf346f05"}
Jul 29 10:58:52.336448 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:52.336417    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk" event={"ID":"6697752f-3304-469e-87b1-05a296014c65","Type":"ContainerStarted","Data":"4b708f6322c1bb8fef717d6c73746cef1a838cd4b3d9f24f6ef47e6f8360d204"}
Jul 29 10:58:52.336631 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:52.336613    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"
Jul 29 10:58:52.354343 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:52.353984    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8" podStartSLOduration=1.831099384 podStartE2EDuration="6.35396492s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.522865536s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:58:52.353178914 +0000 UTC m=+1800.546206264" watchObservedRunningTime="2026-07-29 10:58:52.35396492 +0000 UTC m=+1800.546992269"
Jul 29 10:58:52.363513 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:52.363493    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:52.369585 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:52.369547    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk" podStartSLOduration=2.265739627 podStartE2EDuration="6.369535342s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.103795715s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:58:52.368344367 +0000 UTC m=+1800.561371705" watchObservedRunningTime="2026-07-29 10:58:52.369535342 +0000 UTC m=+1800.562562688"
Jul 29 10:58:56.363134 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:56.363100    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:56.376082 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:56.376046    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:58:57.380229 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:58:57.380196    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:59:23.353139 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:23.353103    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"
Jul 29 10:59:24.423733 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:24.423703    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk_6697752f-3304-469e-87b1-05a296014c65/main/0.log"
Jul 29 10:59:24.586239 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:24.586206    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"]
Jul 29 10:59:24.594970 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:24.594940    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"]
Jul 29 10:59:24.595169 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:24.595147    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk" podUID="6697752f-3304-469e-87b1-05a296014c65" containerName="main" containerID="cri-o://4b708f6322c1bb8fef717d6c73746cef1a838cd4b3d9f24f6ef47e6f8360d204" gracePeriod=30
Jul 29 10:59:24.596881 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:59:24.596842    2584 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:59:24.596881 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:59:24.596881 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:59:24.596881 ip-10-0-140-240 kubenswrapper[2584]:  > containerID="0b46df227effc4e0d4dc404454eeec730d0d308b8dfaece74064c97caf346f05" cmd=["/bin/sleep","15"]
Jul 29 10:59:24.597169 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:24.596898    2584 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8" message="" err=<
Jul 29 10:59:24.597169 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:59:24.597169 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:59:24.597169 ip-10-0-140-240 kubenswrapper[2584]:  >
Jul 29 10:59:24.597169 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:59:24.596911    2584 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:59:24.597169 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:59:24.597169 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 10:59:24.597169 ip-10-0-140-240 kubenswrapper[2584]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8" podUID="321d53a5-474d-4842-8831-abe7afc161e3" containerName="main" containerID="cri-o://0b46df227effc4e0d4dc404454eeec730d0d308b8dfaece74064c97caf346f05"
Jul 29 10:59:24.597169 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:24.596945    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8" podUID="321d53a5-474d-4842-8831-abe7afc161e3" containerName="main" containerID="cri-o://0b46df227effc4e0d4dc404454eeec730d0d308b8dfaece74064c97caf346f05" gracePeriod=60
Jul 29 10:59:24.955104 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:24.955040    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:59:25.044631 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.044599    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-model-cache\") pod \"321d53a5-474d-4842-8831-abe7afc161e3\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") "
Jul 29 10:59:25.044847 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.044647    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/321d53a5-474d-4842-8831-abe7afc161e3-tls-certs\") pod \"321d53a5-474d-4842-8831-abe7afc161e3\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") "
Jul 29 10:59:25.044847 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.044702    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vvhw\" (UniqueName: \"kubernetes.io/projected/321d53a5-474d-4842-8831-abe7afc161e3-kube-api-access-2vvhw\") pod \"321d53a5-474d-4842-8831-abe7afc161e3\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") "
Jul 29 10:59:25.044847 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.044750    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-home\") pod \"321d53a5-474d-4842-8831-abe7afc161e3\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") "
Jul 29 10:59:25.044847 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.044774    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-kserve-provision-location\") pod \"321d53a5-474d-4842-8831-abe7afc161e3\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") "
Jul 29 10:59:25.045057 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.044908    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-dshm\") pod \"321d53a5-474d-4842-8831-abe7afc161e3\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") "
Jul 29 10:59:25.045057 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.044967    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-tmp-dir\") pod \"321d53a5-474d-4842-8831-abe7afc161e3\" (UID: \"321d53a5-474d-4842-8831-abe7afc161e3\") "
Jul 29 10:59:25.045057 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.044919    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "321d53a5-474d-4842-8831-abe7afc161e3" (UID: "321d53a5-474d-4842-8831-abe7afc161e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:59:25.045217 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.045151    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-home" (OuterVolumeSpecName: "home") pod "321d53a5-474d-4842-8831-abe7afc161e3" (UID: "321d53a5-474d-4842-8831-abe7afc161e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:59:25.045320 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.045297    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-model-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.045320 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.045320    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-home\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.045504 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.045403    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "321d53a5-474d-4842-8831-abe7afc161e3" (UID: "321d53a5-474d-4842-8831-abe7afc161e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:59:25.047247 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.047207    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-dshm" (OuterVolumeSpecName: "dshm") pod "321d53a5-474d-4842-8831-abe7afc161e3" (UID: "321d53a5-474d-4842-8831-abe7afc161e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:59:25.047532 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.047509    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321d53a5-474d-4842-8831-abe7afc161e3-kube-api-access-2vvhw" (OuterVolumeSpecName: "kube-api-access-2vvhw") pod "321d53a5-474d-4842-8831-abe7afc161e3" (UID: "321d53a5-474d-4842-8831-abe7afc161e3"). InnerVolumeSpecName "kube-api-access-2vvhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:59:25.047593 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.047545    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/321d53a5-474d-4842-8831-abe7afc161e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "321d53a5-474d-4842-8831-abe7afc161e3" (UID: "321d53a5-474d-4842-8831-abe7afc161e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:59:25.113098 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.113061    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "321d53a5-474d-4842-8831-abe7afc161e3" (UID: "321d53a5-474d-4842-8831-abe7afc161e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:59:25.137138 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.137117    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"
Jul 29 10:59:25.145955 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.145935    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/321d53a5-474d-4842-8831-abe7afc161e3-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.146024 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.145958    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vvhw\" (UniqueName: \"kubernetes.io/projected/321d53a5-474d-4842-8831-abe7afc161e3-kube-api-access-2vvhw\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.146024 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.145969    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-kserve-provision-location\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.146024 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.145977    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-dshm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.146024 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.145985    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/321d53a5-474d-4842-8831-abe7afc161e3-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.247109 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.247008    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6697752f-3304-469e-87b1-05a296014c65-tmp-dir\") pod \"6697752f-3304-469e-87b1-05a296014c65\" (UID: \"6697752f-3304-469e-87b1-05a296014c65\") "
Jul 29 10:59:25.247295 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.247114    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6697752f-3304-469e-87b1-05a296014c65-tls-certs\") pod \"6697752f-3304-469e-87b1-05a296014c65\" (UID: \"6697752f-3304-469e-87b1-05a296014c65\") "
Jul 29 10:59:25.247295 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.247136    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6697752f-3304-469e-87b1-05a296014c65-kserve-provision-location\") pod \"6697752f-3304-469e-87b1-05a296014c65\" (UID: \"6697752f-3304-469e-87b1-05a296014c65\") "
Jul 29 10:59:25.247423 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.247318    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6697752f-3304-469e-87b1-05a296014c65-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6697752f-3304-469e-87b1-05a296014c65" (UID: "6697752f-3304-469e-87b1-05a296014c65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:59:25.249508 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.249460    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6697752f-3304-469e-87b1-05a296014c65-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6697752f-3304-469e-87b1-05a296014c65" (UID: "6697752f-3304-469e-87b1-05a296014c65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:59:25.307874 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.307836    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6697752f-3304-469e-87b1-05a296014c65-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6697752f-3304-469e-87b1-05a296014c65" (UID: "6697752f-3304-469e-87b1-05a296014c65"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:59:25.348605 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.348534    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6697752f-3304-469e-87b1-05a296014c65-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.348605 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.348561    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6697752f-3304-469e-87b1-05a296014c65-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.348605 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.348577    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6697752f-3304-469e-87b1-05a296014c65-kserve-provision-location\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.498382 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.498259    2584 generic.go:356] "Generic (PLEG): container finished" podID="321d53a5-474d-4842-8831-abe7afc161e3" containerID="0b46df227effc4e0d4dc404454eeec730d0d308b8dfaece74064c97caf346f05" exitCode=0
Jul 29 10:59:25.498382 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.498308    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8" event={"ID":"321d53a5-474d-4842-8831-abe7afc161e3","Type":"ContainerDied","Data":"0b46df227effc4e0d4dc404454eeec730d0d308b8dfaece74064c97caf346f05"}
Jul 29 10:59:25.498382 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.498345    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"
Jul 29 10:59:25.498382 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.498363    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8" event={"ID":"321d53a5-474d-4842-8831-abe7afc161e3","Type":"ContainerDied","Data":"d0d8c1125565a1be8c74181a4b9a57b15c54d424feed17a6f736a65f96bcdc21"}
Jul 29 10:59:25.499038 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.498389    2584 scope.go:117] "RemoveContainer" containerID="0b46df227effc4e0d4dc404454eeec730d0d308b8dfaece74064c97caf346f05"
Jul 29 10:59:25.500190 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.500166    2584 generic.go:356] "Generic (PLEG): container finished" podID="6697752f-3304-469e-87b1-05a296014c65" containerID="4b708f6322c1bb8fef717d6c73746cef1a838cd4b3d9f24f6ef47e6f8360d204" exitCode=0
Jul 29 10:59:25.500336 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.500230    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"
Jul 29 10:59:25.500336 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.500238    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk" event={"ID":"6697752f-3304-469e-87b1-05a296014c65","Type":"ContainerDied","Data":"4b708f6322c1bb8fef717d6c73746cef1a838cd4b3d9f24f6ef47e6f8360d204"}
Jul 29 10:59:25.500336 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.500273    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk" event={"ID":"6697752f-3304-469e-87b1-05a296014c65","Type":"ContainerDied","Data":"7dc04947266859231e8c059b55d8fac8df36241542a936931d1e01df93f29b1b"}
Jul 29 10:59:25.511662 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.511636    2584 scope.go:117] "RemoveContainer" containerID="ff2bfe366b5d938c8bc2844433a633c75536972490880b737710c432396bc963"
Jul 29 10:59:25.520574 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.520551    2584 scope.go:117] "RemoveContainer" containerID="0b46df227effc4e0d4dc404454eeec730d0d308b8dfaece74064c97caf346f05"
Jul 29 10:59:25.520887 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:59:25.520860    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b46df227effc4e0d4dc404454eeec730d0d308b8dfaece74064c97caf346f05\": container with ID starting with 0b46df227effc4e0d4dc404454eeec730d0d308b8dfaece74064c97caf346f05 not found: ID does not exist" containerID="0b46df227effc4e0d4dc404454eeec730d0d308b8dfaece74064c97caf346f05"
Jul 29 10:59:25.520973 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.520894    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b46df227effc4e0d4dc404454eeec730d0d308b8dfaece74064c97caf346f05"} err="failed to get container status \"0b46df227effc4e0d4dc404454eeec730d0d308b8dfaece74064c97caf346f05\": rpc error: code = NotFound desc = could not find container \"0b46df227effc4e0d4dc404454eeec730d0d308b8dfaece74064c97caf346f05\": container with ID starting with 0b46df227effc4e0d4dc404454eeec730d0d308b8dfaece74064c97caf346f05 not found: ID does not exist"
Jul 29 10:59:25.520973 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.520914    2584 scope.go:117] "RemoveContainer" containerID="ff2bfe366b5d938c8bc2844433a633c75536972490880b737710c432396bc963"
Jul 29 10:59:25.521196 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:59:25.521178    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2bfe366b5d938c8bc2844433a633c75536972490880b737710c432396bc963\": container with ID starting with ff2bfe366b5d938c8bc2844433a633c75536972490880b737710c432396bc963 not found: ID does not exist" containerID="ff2bfe366b5d938c8bc2844433a633c75536972490880b737710c432396bc963"
Jul 29 10:59:25.521241 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.521202    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2bfe366b5d938c8bc2844433a633c75536972490880b737710c432396bc963"} err="failed to get container status \"ff2bfe366b5d938c8bc2844433a633c75536972490880b737710c432396bc963\": rpc error: code = NotFound desc = could not find container \"ff2bfe366b5d938c8bc2844433a633c75536972490880b737710c432396bc963\": container with ID starting with ff2bfe366b5d938c8bc2844433a633c75536972490880b737710c432396bc963 not found: ID does not exist"
Jul 29 10:59:25.521241 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.521217    2584 scope.go:117] "RemoveContainer" containerID="4b708f6322c1bb8fef717d6c73746cef1a838cd4b3d9f24f6ef47e6f8360d204"
Jul 29 10:59:25.521838 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.521816    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"]
Jul 29 10:59:25.525373 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.525353    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-xmww8"]
Jul 29 10:59:25.529531 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.529515    2584 scope.go:117] "RemoveContainer" containerID="3c5d1020b6fc6a3e3583b6f7f359623c90508155e3da0f9e48aa0c1598ebde24"
Jul 29 10:59:25.537179 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.537153    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"]
Jul 29 10:59:25.537824 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.537808    2584 scope.go:117] "RemoveContainer" containerID="4b708f6322c1bb8fef717d6c73746cef1a838cd4b3d9f24f6ef47e6f8360d204"
Jul 29 10:59:25.538081 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:59:25.538055    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b708f6322c1bb8fef717d6c73746cef1a838cd4b3d9f24f6ef47e6f8360d204\": container with ID starting with 4b708f6322c1bb8fef717d6c73746cef1a838cd4b3d9f24f6ef47e6f8360d204 not found: ID does not exist" containerID="4b708f6322c1bb8fef717d6c73746cef1a838cd4b3d9f24f6ef47e6f8360d204"
Jul 29 10:59:25.538142 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.538084    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b708f6322c1bb8fef717d6c73746cef1a838cd4b3d9f24f6ef47e6f8360d204"} err="failed to get container status \"4b708f6322c1bb8fef717d6c73746cef1a838cd4b3d9f24f6ef47e6f8360d204\": rpc error: code = NotFound desc = could not find container \"4b708f6322c1bb8fef717d6c73746cef1a838cd4b3d9f24f6ef47e6f8360d204\": container with ID starting with 4b708f6322c1bb8fef717d6c73746cef1a838cd4b3d9f24f6ef47e6f8360d204 not found: ID does not exist"
Jul 29 10:59:25.538142 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.538111    2584 scope.go:117] "RemoveContainer" containerID="3c5d1020b6fc6a3e3583b6f7f359623c90508155e3da0f9e48aa0c1598ebde24"
Jul 29 10:59:25.538511 ip-10-0-140-240 kubenswrapper[2584]: E0729 10:59:25.538469    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5d1020b6fc6a3e3583b6f7f359623c90508155e3da0f9e48aa0c1598ebde24\": container with ID starting with 3c5d1020b6fc6a3e3583b6f7f359623c90508155e3da0f9e48aa0c1598ebde24 not found: ID does not exist" containerID="3c5d1020b6fc6a3e3583b6f7f359623c90508155e3da0f9e48aa0c1598ebde24"
Jul 29 10:59:25.538601 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.538507    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5d1020b6fc6a3e3583b6f7f359623c90508155e3da0f9e48aa0c1598ebde24"} err="failed to get container status \"3c5d1020b6fc6a3e3583b6f7f359623c90508155e3da0f9e48aa0c1598ebde24\": rpc error: code = NotFound desc = could not find container \"3c5d1020b6fc6a3e3583b6f7f359623c90508155e3da0f9e48aa0c1598ebde24\": container with ID starting with 3c5d1020b6fc6a3e3583b6f7f359623c90508155e3da0f9e48aa0c1598ebde24 not found: ID does not exist"
Jul 29 10:59:25.540205 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:25.540187    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d78649bb7-8hmnk"]
Jul 29 10:59:26.261103 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:26.261069    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="321d53a5-474d-4842-8831-abe7afc161e3" path="/var/lib/kubelet/pods/321d53a5-474d-4842-8831-abe7afc161e3/volumes"
Jul 29 10:59:26.261727 ip-10-0-140-240 kubenswrapper[2584]: I0729 10:59:26.261702    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6697752f-3304-469e-87b1-05a296014c65" path="/var/lib/kubelet/pods/6697752f-3304-469e-87b1-05a296014c65/volumes"
Jul 29 11:01:34.857929 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.857898    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"]
Jul 29 11:01:34.858537 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.858379    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6697752f-3304-469e-87b1-05a296014c65" containerName="main"
Jul 29 11:01:34.858537 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.858398    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6697752f-3304-469e-87b1-05a296014c65" containerName="main"
Jul 29 11:01:34.858537 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.858414    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="321d53a5-474d-4842-8831-abe7afc161e3" containerName="main"
Jul 29 11:01:34.858537 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.858423    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="321d53a5-474d-4842-8831-abe7afc161e3" containerName="main"
Jul 29 11:01:34.858537 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.858436    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="321d53a5-474d-4842-8831-abe7afc161e3" containerName="storage-initializer"
Jul 29 11:01:34.858537 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.858445    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="321d53a5-474d-4842-8831-abe7afc161e3" containerName="storage-initializer"
Jul 29 11:01:34.858537 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.858461    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6697752f-3304-469e-87b1-05a296014c65" containerName="storage-initializer"
Jul 29 11:01:34.858537 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.858469    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6697752f-3304-469e-87b1-05a296014c65" containerName="storage-initializer"
Jul 29 11:01:34.858971 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.858559    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6697752f-3304-469e-87b1-05a296014c65" containerName="main"
Jul 29 11:01:34.858971 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.858575    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="321d53a5-474d-4842-8831-abe7afc161e3" containerName="main"
Jul 29 11:01:34.860563 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.860544    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"
Jul 29 11:01:34.864758 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.864730    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 29 11:01:34.864901 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.864753    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-9rr5h\""
Jul 29 11:01:34.864901 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.864787    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 29 11:01:34.864901 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.864730    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-gghmp\""
Jul 29 11:01:34.865119 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.865054    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 29 11:01:34.869837 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.869815    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"]
Jul 29 11:01:34.977207 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.977179    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd7vl\" (UniqueName: \"kubernetes.io/projected/9874d62f-f13b-453f-80f2-6b1b54ba413f-kube-api-access-zd7vl\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q\" (UID: \"9874d62f-f13b-453f-80f2-6b1b54ba413f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"
Jul 29 11:01:34.977372 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:34.977227    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9874d62f-f13b-453f-80f2-6b1b54ba413f-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q\" (UID: \"9874d62f-f13b-453f-80f2-6b1b54ba413f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"
Jul 29 11:01:35.078501 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:35.078467    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd7vl\" (UniqueName: \"kubernetes.io/projected/9874d62f-f13b-453f-80f2-6b1b54ba413f-kube-api-access-zd7vl\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q\" (UID: \"9874d62f-f13b-453f-80f2-6b1b54ba413f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"
Jul 29 11:01:35.078739 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:35.078531    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9874d62f-f13b-453f-80f2-6b1b54ba413f-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q\" (UID: \"9874d62f-f13b-453f-80f2-6b1b54ba413f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"
Jul 29 11:01:35.081112 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:35.081083    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9874d62f-f13b-453f-80f2-6b1b54ba413f-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q\" (UID: \"9874d62f-f13b-453f-80f2-6b1b54ba413f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"
Jul 29 11:01:35.087845 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:35.087820    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd7vl\" (UniqueName: \"kubernetes.io/projected/9874d62f-f13b-453f-80f2-6b1b54ba413f-kube-api-access-zd7vl\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q\" (UID: \"9874d62f-f13b-453f-80f2-6b1b54ba413f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"
Jul 29 11:01:35.171664 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:35.171567    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"
Jul 29 11:01:35.298804 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:35.298773    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"]
Jul 29 11:01:35.301320 ip-10-0-140-240 kubenswrapper[2584]: W0729 11:01:35.301291    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9874d62f_f13b_453f_80f2_6b1b54ba413f.slice/crio-2cd4bdd1532621dba7de475636e417d2e15fd05477085cca417404e703f4c079 WatchSource:0}: Error finding container 2cd4bdd1532621dba7de475636e417d2e15fd05477085cca417404e703f4c079: Status 404 returned error can't find the container with id 2cd4bdd1532621dba7de475636e417d2e15fd05477085cca417404e703f4c079
Jul 29 11:01:36.055486 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:36.055447    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q" event={"ID":"9874d62f-f13b-453f-80f2-6b1b54ba413f","Type":"ContainerStarted","Data":"285fb890a0b3e39be66091a526355bf92e423154a42da0f494bc7269a526f734"}
Jul 29 11:01:36.055908 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:36.055510    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q" event={"ID":"9874d62f-f13b-453f-80f2-6b1b54ba413f","Type":"ContainerStarted","Data":"2cd4bdd1532621dba7de475636e417d2e15fd05477085cca417404e703f4c079"}
Jul 29 11:01:36.055908 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:36.055541    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"
Jul 29 11:01:36.072836 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:01:36.072788    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q" podStartSLOduration=2.072772822 podStartE2EDuration="2.072772822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:01:36.071293972 +0000 UTC m=+1964.264321319" watchObservedRunningTime="2026-07-29 11:01:36.072772822 +0000 UTC m=+1964.265800168"
Jul 29 11:02:07.062175 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:07.062144    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"
Jul 29 11:02:08.134392 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:08.134356    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"]
Jul 29 11:02:17.062150 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:17.062106    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q" podUID="9874d62f-f13b-453f-80f2-6b1b54ba413f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:02:23.153348 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:23.153309    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q" podUID="9874d62f-f13b-453f-80f2-6b1b54ba413f" containerName="main" containerID="cri-o://285fb890a0b3e39be66091a526355bf92e423154a42da0f494bc7269a526f734" gracePeriod=45
Jul 29 11:02:23.270461 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:23.270432    2584 generic.go:356] "Generic (PLEG): container finished" podID="9874d62f-f13b-453f-80f2-6b1b54ba413f" containerID="285fb890a0b3e39be66091a526355bf92e423154a42da0f494bc7269a526f734" exitCode=0
Jul 29 11:02:23.270613 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:23.270504    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q" event={"ID":"9874d62f-f13b-453f-80f2-6b1b54ba413f","Type":"ContainerDied","Data":"285fb890a0b3e39be66091a526355bf92e423154a42da0f494bc7269a526f734"}
Jul 29 11:02:23.390438 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:23.390408    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"
Jul 29 11:02:23.490146 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:23.490057    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd7vl\" (UniqueName: \"kubernetes.io/projected/9874d62f-f13b-453f-80f2-6b1b54ba413f-kube-api-access-zd7vl\") pod \"9874d62f-f13b-453f-80f2-6b1b54ba413f\" (UID: \"9874d62f-f13b-453f-80f2-6b1b54ba413f\") "
Jul 29 11:02:23.490146 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:23.490100    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9874d62f-f13b-453f-80f2-6b1b54ba413f-tls-certs\") pod \"9874d62f-f13b-453f-80f2-6b1b54ba413f\" (UID: \"9874d62f-f13b-453f-80f2-6b1b54ba413f\") "
Jul 29 11:02:23.492370 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:23.492346    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9874d62f-f13b-453f-80f2-6b1b54ba413f-kube-api-access-zd7vl" (OuterVolumeSpecName: "kube-api-access-zd7vl") pod "9874d62f-f13b-453f-80f2-6b1b54ba413f" (UID: "9874d62f-f13b-453f-80f2-6b1b54ba413f"). InnerVolumeSpecName "kube-api-access-zd7vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:02:23.492443 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:23.492347    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9874d62f-f13b-453f-80f2-6b1b54ba413f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9874d62f-f13b-453f-80f2-6b1b54ba413f" (UID: "9874d62f-f13b-453f-80f2-6b1b54ba413f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:02:23.591056 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:23.591015    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd7vl\" (UniqueName: \"kubernetes.io/projected/9874d62f-f13b-453f-80f2-6b1b54ba413f-kube-api-access-zd7vl\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:02:23.591056 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:23.591050    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9874d62f-f13b-453f-80f2-6b1b54ba413f-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:02:24.276275 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:24.276252    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"
Jul 29 11:02:24.276275 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:24.276267    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q" event={"ID":"9874d62f-f13b-453f-80f2-6b1b54ba413f","Type":"ContainerDied","Data":"2cd4bdd1532621dba7de475636e417d2e15fd05477085cca417404e703f4c079"}
Jul 29 11:02:24.276739 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:24.276321    2584 scope.go:117] "RemoveContainer" containerID="285fb890a0b3e39be66091a526355bf92e423154a42da0f494bc7269a526f734"
Jul 29 11:02:24.292422 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:24.292394    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"]
Jul 29 11:02:24.298590 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:24.298568    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche9ml9q"]
Jul 29 11:02:26.254652 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:02:26.254619    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9874d62f-f13b-453f-80f2-6b1b54ba413f" path="/var/lib/kubelet/pods/9874d62f-f13b-453f-80f2-6b1b54ba413f/volumes"
Jul 29 11:03:39.920744 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:39.920710    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"]
Jul 29 11:03:39.921323 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:39.921045    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9874d62f-f13b-453f-80f2-6b1b54ba413f" containerName="main"
Jul 29 11:03:39.921323 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:39.921055    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9874d62f-f13b-453f-80f2-6b1b54ba413f" containerName="main"
Jul 29 11:03:39.921323 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:39.921107    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9874d62f-f13b-453f-80f2-6b1b54ba413f" containerName="main"
Jul 29 11:03:39.922927 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:39.922907    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"
Jul 29 11:03:39.927224 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:39.927178    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 29 11:03:39.927343 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:39.927286    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 29 11:03:39.927343 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:39.927293    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 29 11:03:39.927343 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:39.927292    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-xr7ht\""
Jul 29 11:03:39.927515 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:39.927355    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-2rqx9\""
Jul 29 11:03:39.939530 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:39.939506    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"]
Jul 29 11:03:40.019553 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:40.019459    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217c408b-993e-41d2-b963-846faaa45e60-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt\" (UID: \"217c408b-993e-41d2-b963-846faaa45e60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"
Jul 29 11:03:40.019553 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:40.019504    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzfkk\" (UniqueName: \"kubernetes.io/projected/217c408b-993e-41d2-b963-846faaa45e60-kube-api-access-hzfkk\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt\" (UID: \"217c408b-993e-41d2-b963-846faaa45e60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"
Jul 29 11:03:40.120904 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:40.120860    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217c408b-993e-41d2-b963-846faaa45e60-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt\" (UID: \"217c408b-993e-41d2-b963-846faaa45e60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"
Jul 29 11:03:40.121090 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:40.120920    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzfkk\" (UniqueName: \"kubernetes.io/projected/217c408b-993e-41d2-b963-846faaa45e60-kube-api-access-hzfkk\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt\" (UID: \"217c408b-993e-41d2-b963-846faaa45e60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"
Jul 29 11:03:40.123377 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:40.123353    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217c408b-993e-41d2-b963-846faaa45e60-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt\" (UID: \"217c408b-993e-41d2-b963-846faaa45e60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"
Jul 29 11:03:40.128501 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:40.128475    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzfkk\" (UniqueName: \"kubernetes.io/projected/217c408b-993e-41d2-b963-846faaa45e60-kube-api-access-hzfkk\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt\" (UID: \"217c408b-993e-41d2-b963-846faaa45e60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"
Jul 29 11:03:40.236438 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:40.236399    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"
Jul 29 11:03:40.364261 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:40.364229    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"]
Jul 29 11:03:40.367491 ip-10-0-140-240 kubenswrapper[2584]: W0729 11:03:40.367461    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217c408b_993e_41d2_b963_846faaa45e60.slice/crio-4fa356b7da001cecddeba6c1f20ffe51fa954bcdcff289bc3f8e880cd0e1987e WatchSource:0}: Error finding container 4fa356b7da001cecddeba6c1f20ffe51fa954bcdcff289bc3f8e880cd0e1987e: Status 404 returned error can't find the container with id 4fa356b7da001cecddeba6c1f20ffe51fa954bcdcff289bc3f8e880cd0e1987e
Jul 29 11:03:40.608672 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:40.608580    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt" event={"ID":"217c408b-993e-41d2-b963-846faaa45e60","Type":"ContainerStarted","Data":"21a9c32e796ed4f8c6e44a527d18ba4476b9a4eb2a8f0a2ffb15c74a1812ecfd"}
Jul 29 11:03:40.608672 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:40.608628    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt" event={"ID":"217c408b-993e-41d2-b963-846faaa45e60","Type":"ContainerStarted","Data":"4fa356b7da001cecddeba6c1f20ffe51fa954bcdcff289bc3f8e880cd0e1987e"}
Jul 29 11:03:40.608913 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:40.608712    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"
Jul 29 11:03:40.625735 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:03:40.625627    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt" podStartSLOduration=1.625607752 podStartE2EDuration="1.625607752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:03:40.624587691 +0000 UTC m=+2088.817615053" watchObservedRunningTime="2026-07-29 11:03:40.625607752 +0000 UTC m=+2088.818635102"
Jul 29 11:04:11.615840 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:11.615811    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"
Jul 29 11:04:12.849229 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:12.849195    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"]
Jul 29 11:04:21.615718 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:21.615660    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt" podUID="217c408b-993e-41d2-b963-846faaa45e60" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:04:27.868312 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:27.868261    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt" podUID="217c408b-993e-41d2-b963-846faaa45e60" containerName="main" containerID="cri-o://21a9c32e796ed4f8c6e44a527d18ba4476b9a4eb2a8f0a2ffb15c74a1812ecfd" gracePeriod=45
Jul 29 11:04:27.906541 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:04:27.906502    2584 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217c408b_993e_41d2_b963_846faaa45e60.slice/crio-conmon-21a9c32e796ed4f8c6e44a527d18ba4476b9a4eb2a8f0a2ffb15c74a1812ecfd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217c408b_993e_41d2_b963_846faaa45e60.slice/crio-21a9c32e796ed4f8c6e44a527d18ba4476b9a4eb2a8f0a2ffb15c74a1812ecfd.scope\": RecentStats: unable to find data in memory cache]"
Jul 29 11:04:28.118302 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.118279    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"
Jul 29 11:04:28.251171 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.251083    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzfkk\" (UniqueName: \"kubernetes.io/projected/217c408b-993e-41d2-b963-846faaa45e60-kube-api-access-hzfkk\") pod \"217c408b-993e-41d2-b963-846faaa45e60\" (UID: \"217c408b-993e-41d2-b963-846faaa45e60\") "
Jul 29 11:04:28.251171 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.251164    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217c408b-993e-41d2-b963-846faaa45e60-tls-certs\") pod \"217c408b-993e-41d2-b963-846faaa45e60\" (UID: \"217c408b-993e-41d2-b963-846faaa45e60\") "
Jul 29 11:04:28.253404 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.253375    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/217c408b-993e-41d2-b963-846faaa45e60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "217c408b-993e-41d2-b963-846faaa45e60" (UID: "217c408b-993e-41d2-b963-846faaa45e60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:04:28.253546 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.253476    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217c408b-993e-41d2-b963-846faaa45e60-kube-api-access-hzfkk" (OuterVolumeSpecName: "kube-api-access-hzfkk") pod "217c408b-993e-41d2-b963-846faaa45e60" (UID: "217c408b-993e-41d2-b963-846faaa45e60"). InnerVolumeSpecName "kube-api-access-hzfkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:04:28.352773 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.352733    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzfkk\" (UniqueName: \"kubernetes.io/projected/217c408b-993e-41d2-b963-846faaa45e60-kube-api-access-hzfkk\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:04:28.352773 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.352763    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217c408b-993e-41d2-b963-846faaa45e60-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:04:28.824966 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.824936    2584 generic.go:356] "Generic (PLEG): container finished" podID="217c408b-993e-41d2-b963-846faaa45e60" containerID="21a9c32e796ed4f8c6e44a527d18ba4476b9a4eb2a8f0a2ffb15c74a1812ecfd" exitCode=0
Jul 29 11:04:28.825148 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.824979    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"
Jul 29 11:04:28.825148 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.824997    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt" event={"ID":"217c408b-993e-41d2-b963-846faaa45e60","Type":"ContainerDied","Data":"21a9c32e796ed4f8c6e44a527d18ba4476b9a4eb2a8f0a2ffb15c74a1812ecfd"}
Jul 29 11:04:28.825148 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.825049    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt" event={"ID":"217c408b-993e-41d2-b963-846faaa45e60","Type":"ContainerDied","Data":"4fa356b7da001cecddeba6c1f20ffe51fa954bcdcff289bc3f8e880cd0e1987e"}
Jul 29 11:04:28.825148 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.825079    2584 scope.go:117] "RemoveContainer" containerID="21a9c32e796ed4f8c6e44a527d18ba4476b9a4eb2a8f0a2ffb15c74a1812ecfd"
Jul 29 11:04:28.834044 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.834025    2584 scope.go:117] "RemoveContainer" containerID="21a9c32e796ed4f8c6e44a527d18ba4476b9a4eb2a8f0a2ffb15c74a1812ecfd"
Jul 29 11:04:28.834319 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:04:28.834298    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a9c32e796ed4f8c6e44a527d18ba4476b9a4eb2a8f0a2ffb15c74a1812ecfd\": container with ID starting with 21a9c32e796ed4f8c6e44a527d18ba4476b9a4eb2a8f0a2ffb15c74a1812ecfd not found: ID does not exist" containerID="21a9c32e796ed4f8c6e44a527d18ba4476b9a4eb2a8f0a2ffb15c74a1812ecfd"
Jul 29 11:04:28.834362 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.834333    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a9c32e796ed4f8c6e44a527d18ba4476b9a4eb2a8f0a2ffb15c74a1812ecfd"} err="failed to get container status \"21a9c32e796ed4f8c6e44a527d18ba4476b9a4eb2a8f0a2ffb15c74a1812ecfd\": rpc error: code = NotFound desc = could not find container \"21a9c32e796ed4f8c6e44a527d18ba4476b9a4eb2a8f0a2ffb15c74a1812ecfd\": container with ID starting with 21a9c32e796ed4f8c6e44a527d18ba4476b9a4eb2a8f0a2ffb15c74a1812ecfd not found: ID does not exist"
Jul 29 11:04:28.842199 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.842164    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"]
Jul 29 11:04:28.843546 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:28.843525    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8vpqt"]
Jul 29 11:04:30.255069 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:04:30.255029    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="217c408b-993e-41d2-b963-846faaa45e60" path="/var/lib/kubelet/pods/217c408b-993e-41d2-b963-846faaa45e60/volumes"
Jul 29 11:08:28.186780 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:28.186741    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"]
Jul 29 11:08:28.199121 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:08:28.199074    2584 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 11:08:28.199121 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:08:28.199121 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 11:08:28.199121 ip-10-0-140-240 kubenswrapper[2584]:  > containerID="ebffb5a0d76549b6cbe1f56c476b0f3c318cc6ef2689b17f21b7eef8078a4679" cmd=["/bin/sleep","15"]
Jul 29 11:08:28.199502 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:28.199146    2584 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k" message="" err=<
Jul 29 11:08:28.199502 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:08:28.199502 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 11:08:28.199502 ip-10-0-140-240 kubenswrapper[2584]:  >
Jul 29 11:08:28.199502 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:08:28.199166    2584 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 11:08:28.199502 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:08:28.199502 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 11:08:28.199502 ip-10-0-140-240 kubenswrapper[2584]:  > pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k" podUID="7a9be8c2-2468-4164-a218-8e9322ae9db9" containerName="main" containerID="cri-o://ebffb5a0d76549b6cbe1f56c476b0f3c318cc6ef2689b17f21b7eef8078a4679"
Jul 29 11:08:28.199502 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:28.199230    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k" podUID="7a9be8c2-2468-4164-a218-8e9322ae9db9" containerName="main" containerID="cri-o://ebffb5a0d76549b6cbe1f56c476b0f3c318cc6ef2689b17f21b7eef8078a4679" gracePeriod=60
Jul 29 11:08:28.817158 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:28.817119    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k" podUID="7a9be8c2-2468-4164-a218-8e9322ae9db9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 29 11:08:28.898970 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:28.898940    2584 generic.go:356] "Generic (PLEG): container finished" podID="7a9be8c2-2468-4164-a218-8e9322ae9db9" containerID="ebffb5a0d76549b6cbe1f56c476b0f3c318cc6ef2689b17f21b7eef8078a4679" exitCode=0
Jul 29 11:08:28.899146 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:28.899020    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k" event={"ID":"7a9be8c2-2468-4164-a218-8e9322ae9db9","Type":"ContainerDied","Data":"ebffb5a0d76549b6cbe1f56c476b0f3c318cc6ef2689b17f21b7eef8078a4679"}
Jul 29 11:08:28.950270 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:28.950245    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 11:08:29.080013 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.079921    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pndn\" (UniqueName: \"kubernetes.io/projected/7a9be8c2-2468-4164-a218-8e9322ae9db9-kube-api-access-2pndn\") pod \"7a9be8c2-2468-4164-a218-8e9322ae9db9\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") "
Jul 29 11:08:29.080013 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.079970    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-home\") pod \"7a9be8c2-2468-4164-a218-8e9322ae9db9\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") "
Jul 29 11:08:29.080236 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.080014    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a9be8c2-2468-4164-a218-8e9322ae9db9-tls-certs\") pod \"7a9be8c2-2468-4164-a218-8e9322ae9db9\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") "
Jul 29 11:08:29.080236 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.080073    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-model-cache\") pod \"7a9be8c2-2468-4164-a218-8e9322ae9db9\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") "
Jul 29 11:08:29.080236 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.080096    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-tmp-dir\") pod \"7a9be8c2-2468-4164-a218-8e9322ae9db9\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") "
Jul 29 11:08:29.080236 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.080149    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-dshm\") pod \"7a9be8c2-2468-4164-a218-8e9322ae9db9\" (UID: \"7a9be8c2-2468-4164-a218-8e9322ae9db9\") "
Jul 29 11:08:29.080450 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.080296    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-home" (OuterVolumeSpecName: "home") pod "7a9be8c2-2468-4164-a218-8e9322ae9db9" (UID: "7a9be8c2-2468-4164-a218-8e9322ae9db9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:08:29.080450 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.080379    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a9be8c2-2468-4164-a218-8e9322ae9db9" (UID: "7a9be8c2-2468-4164-a218-8e9322ae9db9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:08:29.080450 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.080396    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a9be8c2-2468-4164-a218-8e9322ae9db9" (UID: "7a9be8c2-2468-4164-a218-8e9322ae9db9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:08:29.080450 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.080440    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-home\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:08:29.080599 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.080455    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:08:29.082275 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.082246    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-dshm" (OuterVolumeSpecName: "dshm") pod "7a9be8c2-2468-4164-a218-8e9322ae9db9" (UID: "7a9be8c2-2468-4164-a218-8e9322ae9db9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:08:29.082402 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.082329    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9be8c2-2468-4164-a218-8e9322ae9db9-kube-api-access-2pndn" (OuterVolumeSpecName: "kube-api-access-2pndn") pod "7a9be8c2-2468-4164-a218-8e9322ae9db9" (UID: "7a9be8c2-2468-4164-a218-8e9322ae9db9"). InnerVolumeSpecName "kube-api-access-2pndn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:08:29.082469 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.082410    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a9be8c2-2468-4164-a218-8e9322ae9db9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a9be8c2-2468-4164-a218-8e9322ae9db9" (UID: "7a9be8c2-2468-4164-a218-8e9322ae9db9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:08:29.181034 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.181006    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a9be8c2-2468-4164-a218-8e9322ae9db9-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:08:29.181034 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.181031    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-model-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:08:29.181034 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.181039    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a9be8c2-2468-4164-a218-8e9322ae9db9-dshm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:08:29.181268 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.181047    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pndn\" (UniqueName: \"kubernetes.io/projected/7a9be8c2-2468-4164-a218-8e9322ae9db9-kube-api-access-2pndn\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:08:29.904975 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.904944    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"
Jul 29 11:08:29.905440 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.904943    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k" event={"ID":"7a9be8c2-2468-4164-a218-8e9322ae9db9","Type":"ContainerDied","Data":"1acaf20d908acd9abf4465a4ebc3ca969c5cd9d9fcef2401505228266ad0a3ec"}
Jul 29 11:08:29.905440 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.905080    2584 scope.go:117] "RemoveContainer" containerID="ebffb5a0d76549b6cbe1f56c476b0f3c318cc6ef2689b17f21b7eef8078a4679"
Jul 29 11:08:29.926379 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.926342    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"]
Jul 29 11:08:29.932421 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:29.932392    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-k8m4k"]
Jul 29 11:08:30.254614 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:30.254528    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a9be8c2-2468-4164-a218-8e9322ae9db9" path="/var/lib/kubelet/pods/7a9be8c2-2468-4164-a218-8e9322ae9db9/volumes"
Jul 29 11:08:59.024728 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.024696    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"]
Jul 29 11:08:59.025130 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.025024    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a9be8c2-2468-4164-a218-8e9322ae9db9" containerName="main"
Jul 29 11:08:59.025130 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.025035    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a9be8c2-2468-4164-a218-8e9322ae9db9" containerName="main"
Jul 29 11:08:59.025130 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.025048    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="217c408b-993e-41d2-b963-846faaa45e60" containerName="main"
Jul 29 11:08:59.025130 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.025054    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="217c408b-993e-41d2-b963-846faaa45e60" containerName="main"
Jul 29 11:08:59.025130 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.025108    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a9be8c2-2468-4164-a218-8e9322ae9db9" containerName="main"
Jul 29 11:08:59.025130 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.025119    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="217c408b-993e-41d2-b963-846faaa45e60" containerName="main"
Jul 29 11:08:59.027955 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.027933    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.032009 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.031985    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"kube-root-ca.crt\""
Jul 29 11:08:59.032009 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.032003    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-kserve-self-signed-certs\""
Jul 29 11:08:59.032215 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.031989    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"openshift-service-ca.crt\""
Jul 29 11:08:59.032215 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.032057    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"default-dockercfg-rsqwv\""
Jul 29 11:08:59.038004 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.037980    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"]
Jul 29 11:08:59.128489 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.128461    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.128728 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.128504    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.128728 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.128574    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.128728 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.128620    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.128728 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.128670    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.128728 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.128714    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz6l5\" (UniqueName: \"kubernetes.io/projected/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-kube-api-access-sz6l5\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.229913 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.229876    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.229913 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.229918    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.230169 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.229947    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.230169 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.229964    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz6l5\" (UniqueName: \"kubernetes.io/projected/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-kube-api-access-sz6l5\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.230169 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.229999    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.230169 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.230032    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.230449 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.230430    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.230497 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.230481    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.230544 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.230511    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.232357 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.232333    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.232497 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.232478    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.237748 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.237726    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz6l5\" (UniqueName: \"kubernetes.io/projected/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-kube-api-access-sz6l5\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.338754 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.338723    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:08:59.469789 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:08:59.469753    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"]
Jul 29 11:08:59.472304 ip-10-0-140-240 kubenswrapper[2584]: W0729 11:08:59.472280    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f693c0_ff0c_4b96_8c8a_dd9b1edd06ac.slice/crio-34e5b2f6c39c87b8b667c4797e3201b62be2d0e42e9b40397289f0c0fa396778 WatchSource:0}: Error finding container 34e5b2f6c39c87b8b667c4797e3201b62be2d0e42e9b40397289f0c0fa396778: Status 404 returned error can't find the container with id 34e5b2f6c39c87b8b667c4797e3201b62be2d0e42e9b40397289f0c0fa396778
Jul 29 11:09:00.036363 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:09:00.036330    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl" event={"ID":"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac","Type":"ContainerStarted","Data":"ca0a45f180f9c787cc8cc045be89ecd0912780e3e2031583a38c5b6aeb840135"}
Jul 29 11:09:00.036928 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:09:00.036384    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl" event={"ID":"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac","Type":"ContainerStarted","Data":"34e5b2f6c39c87b8b667c4797e3201b62be2d0e42e9b40397289f0c0fa396778"}
Jul 29 11:09:00.075949 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:09:00.075893    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl" podStartSLOduration=1.075877392 podStartE2EDuration="1.075877392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:09:00.074890046 +0000 UTC m=+2408.267917394" watchObservedRunningTime="2026-07-29 11:09:00.075877392 +0000 UTC m=+2408.268904739"
Jul 29 11:09:00.339139 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:09:00.339052    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:09:09.339516 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:09:09.339471    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:09:09.352294 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:09:09.352265    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:09:10.093733 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:09:10.093703    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:12:04.083874 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.083838    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd"]
Jul 29 11:12:04.087263 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.087244    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd"
Jul 29 11:12:04.092562 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.092540    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 29 11:12:04.093995 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.093966    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 29 11:12:04.094131 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.093966    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-mcnx5\""
Jul 29 11:12:04.094254 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.094225    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 29 11:12:04.094475 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.094457    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-p7mgs\""
Jul 29 11:12:04.098608 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.098582    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd"]
Jul 29 11:12:04.130457 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.130420    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snncn\" (UniqueName: \"kubernetes.io/projected/701ba11c-e9ed-4ead-9b4f-39ec15efcd62-kube-api-access-snncn\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd\" (UID: \"701ba11c-e9ed-4ead-9b4f-39ec15efcd62\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd"
Jul 29 11:12:04.130667 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.130470    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701ba11c-e9ed-4ead-9b4f-39ec15efcd62-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd\" (UID: \"701ba11c-e9ed-4ead-9b4f-39ec15efcd62\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd"
Jul 29 11:12:04.231250 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.231216    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snncn\" (UniqueName: \"kubernetes.io/projected/701ba11c-e9ed-4ead-9b4f-39ec15efcd62-kube-api-access-snncn\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd\" (UID: \"701ba11c-e9ed-4ead-9b4f-39ec15efcd62\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd"
Jul 29 11:12:04.231446 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.231262    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701ba11c-e9ed-4ead-9b4f-39ec15efcd62-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd\" (UID: \"701ba11c-e9ed-4ead-9b4f-39ec15efcd62\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd"
Jul 29 11:12:04.233831 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.233806    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701ba11c-e9ed-4ead-9b4f-39ec15efcd62-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd\" (UID: \"701ba11c-e9ed-4ead-9b4f-39ec15efcd62\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd"
Jul 29 11:12:04.239552 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.239517    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snncn\" (UniqueName: \"kubernetes.io/projected/701ba11c-e9ed-4ead-9b4f-39ec15efcd62-kube-api-access-snncn\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd\" (UID: \"701ba11c-e9ed-4ead-9b4f-39ec15efcd62\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd"
Jul 29 11:12:04.400907 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.400812    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd"
Jul 29 11:12:04.534436 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.534410    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd"]
Jul 29 11:12:04.536800 ip-10-0-140-240 kubenswrapper[2584]: W0729 11:12:04.536770    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701ba11c_e9ed_4ead_9b4f_39ec15efcd62.slice/crio-725f280e137f0a8974555121726b5c12c89a6d3560de2aa27363a5dc02c6fe04 WatchSource:0}: Error finding container 725f280e137f0a8974555121726b5c12c89a6d3560de2aa27363a5dc02c6fe04: Status 404 returned error can't find the container with id 725f280e137f0a8974555121726b5c12c89a6d3560de2aa27363a5dc02c6fe04
Jul 29 11:12:04.801187 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.801152    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd" event={"ID":"701ba11c-e9ed-4ead-9b4f-39ec15efcd62","Type":"ContainerStarted","Data":"293e7d4ea2b49e05abc2d09d4b79ab8ce550056d0a684ab3c79924c3ffabee8c"}
Jul 29 11:12:04.801389 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.801203    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd" event={"ID":"701ba11c-e9ed-4ead-9b4f-39ec15efcd62","Type":"ContainerStarted","Data":"725f280e137f0a8974555121726b5c12c89a6d3560de2aa27363a5dc02c6fe04"}
Jul 29 11:12:04.801389 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.801251    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd"
Jul 29 11:12:04.823672 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:04.823605    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd" podStartSLOduration=0.823584514 podStartE2EDuration="823.584514ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:12:04.821049869 +0000 UTC m=+2593.014077217" watchObservedRunningTime="2026-07-29 11:12:04.823584514 +0000 UTC m=+2593.016611863"
Jul 29 11:12:35.808193 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:12:35.808113    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd"
Jul 29 11:23:59.995249 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:23:59.995215    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"]
Jul 29 11:24:00.004090 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:24:00.004052    2584 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 11:24:00.004090 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:24:00.004090 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 11:24:00.004090 ip-10-0-140-240 kubenswrapper[2584]:  > containerID="ca0a45f180f9c787cc8cc045be89ecd0912780e3e2031583a38c5b6aeb840135" cmd=["/bin/sleep","15"]
Jul 29 11:24:00.004433 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.004102    2584 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl" message="" err=<
Jul 29 11:24:00.004433 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:24:00.004433 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 11:24:00.004433 ip-10-0-140-240 kubenswrapper[2584]:  >
Jul 29 11:24:00.004433 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:24:00.004116    2584 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 11:24:00.004433 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:24:00.004433 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 11:24:00.004433 ip-10-0-140-240 kubenswrapper[2584]:  > pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl" podUID="89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac" containerName="main" containerID="cri-o://ca0a45f180f9c787cc8cc045be89ecd0912780e3e2031583a38c5b6aeb840135"
Jul 29 11:24:00.004433 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.004151    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl" podUID="89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac" containerName="main" containerID="cri-o://ca0a45f180f9c787cc8cc045be89ecd0912780e3e2031583a38c5b6aeb840135" gracePeriod=60
Jul 29 11:24:00.083442 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.083402    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl" podUID="89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 29 11:24:00.757896 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.757873    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:24:00.868120 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.868032    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-home\") pod \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") "
Jul 29 11:24:00.868120 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.868091    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-tls-certs\") pod \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") "
Jul 29 11:24:00.868120 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.868118    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-tmp-dir\") pod \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") "
Jul 29 11:24:00.868415 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.868142    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz6l5\" (UniqueName: \"kubernetes.io/projected/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-kube-api-access-sz6l5\") pod \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") "
Jul 29 11:24:00.868415 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.868177    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-model-cache\") pod \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") "
Jul 29 11:24:00.868415 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.868203    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-dshm\") pod \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\" (UID: \"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac\") "
Jul 29 11:24:00.868415 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.868329    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-home" (OuterVolumeSpecName: "home") pod "89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac" (UID: "89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:24:00.868415 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.868395    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac" (UID: "89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:24:00.868614 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.868417    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac" (UID: "89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:24:00.868614 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.868449    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-home\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:24:00.870520 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.870488    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-kube-api-access-sz6l5" (OuterVolumeSpecName: "kube-api-access-sz6l5") pod "89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac" (UID: "89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac"). InnerVolumeSpecName "kube-api-access-sz6l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:24:00.870520 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.870493    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac" (UID: "89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:24:00.870520 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.870493    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-dshm" (OuterVolumeSpecName: "dshm") pod "89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac" (UID: "89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:24:00.969549 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.969512    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:24:00.969549 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.969542    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:24:00.969549 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.969553    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz6l5\" (UniqueName: \"kubernetes.io/projected/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-kube-api-access-sz6l5\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:24:00.969822 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.969562    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-model-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:24:00.969822 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:00.969573    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac-dshm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:24:01.021016 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:01.020980    2584 generic.go:356] "Generic (PLEG): container finished" podID="89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac" containerID="ca0a45f180f9c787cc8cc045be89ecd0912780e3e2031583a38c5b6aeb840135" exitCode=0
Jul 29 11:24:01.021414 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:01.021028    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"
Jul 29 11:24:01.021414 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:01.021073    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl" event={"ID":"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac","Type":"ContainerDied","Data":"ca0a45f180f9c787cc8cc045be89ecd0912780e3e2031583a38c5b6aeb840135"}
Jul 29 11:24:01.021523 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:01.021430    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl" event={"ID":"89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac","Type":"ContainerDied","Data":"34e5b2f6c39c87b8b667c4797e3201b62be2d0e42e9b40397289f0c0fa396778"}
Jul 29 11:24:01.021523 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:01.021500    2584 scope.go:117] "RemoveContainer" containerID="ca0a45f180f9c787cc8cc045be89ecd0912780e3e2031583a38c5b6aeb840135"
Jul 29 11:24:01.036971 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:01.036948    2584 scope.go:117] "RemoveContainer" containerID="ca0a45f180f9c787cc8cc045be89ecd0912780e3e2031583a38c5b6aeb840135"
Jul 29 11:24:01.037241 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:24:01.037219    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0a45f180f9c787cc8cc045be89ecd0912780e3e2031583a38c5b6aeb840135\": container with ID starting with ca0a45f180f9c787cc8cc045be89ecd0912780e3e2031583a38c5b6aeb840135 not found: ID does not exist" containerID="ca0a45f180f9c787cc8cc045be89ecd0912780e3e2031583a38c5b6aeb840135"
Jul 29 11:24:01.037288 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:01.037252    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0a45f180f9c787cc8cc045be89ecd0912780e3e2031583a38c5b6aeb840135"} err="failed to get container status \"ca0a45f180f9c787cc8cc045be89ecd0912780e3e2031583a38c5b6aeb840135\": rpc error: code = NotFound desc = could not find container \"ca0a45f180f9c787cc8cc045be89ecd0912780e3e2031583a38c5b6aeb840135\": container with ID starting with ca0a45f180f9c787cc8cc045be89ecd0912780e3e2031583a38c5b6aeb840135 not found: ID does not exist"
Jul 29 11:24:01.045786 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:01.045761    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"]
Jul 29 11:24:01.049270 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:01.049243    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-82vxl"]
Jul 29 11:24:02.255089 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:02.255057    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac" path="/var/lib/kubelet/pods/89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac/volumes"
Jul 29 11:24:36.055437 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.055352    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 29 11:24:36.056118 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.055756    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac" containerName="main"
Jul 29 11:24:36.056118 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.055769    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac" containerName="main"
Jul 29 11:24:36.056118 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.055821    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f693c0-ff0c-4b96-8c8a-dd9b1edd06ac" containerName="main"
Jul 29 11:24:36.058665 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.058649    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.061433 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.061414    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 29 11:24:36.062797 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.062776    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-29cvz\""
Jul 29 11:24:36.062797 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.062788    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 29 11:24:36.062978 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.062779    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 29 11:24:36.062978 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.062830    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-mn-dockercfg-nrqcq\""
Jul 29 11:24:36.068736 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.068715    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 29 11:24:36.077727 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.077671    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/926c386f-291b-4d77-b823-38865ee5fc9e-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.077845 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.077747    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-home\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.077845 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.077790    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.077845 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.077811    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.077958 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.077869    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.077958 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.077899    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc5gm\" (UniqueName: \"kubernetes.io/projected/926c386f-291b-4d77-b823-38865ee5fc9e-kube-api-access-rc5gm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.178257 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.178220    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-home\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.178257 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.178264    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.178526 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.178286    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.178526 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.178311    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.178526 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.178333    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc5gm\" (UniqueName: \"kubernetes.io/projected/926c386f-291b-4d77-b823-38865ee5fc9e-kube-api-access-rc5gm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.178526 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.178362    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/926c386f-291b-4d77-b823-38865ee5fc9e-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.178764 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.178651    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-home\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.178764 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.178714    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.178859 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.178763    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.180918 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.180853    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/926c386f-291b-4d77-b823-38865ee5fc9e-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.180918 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.180905    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.188840 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.188819    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc5gm\" (UniqueName: \"kubernetes.io/projected/926c386f-291b-4d77-b823-38865ee5fc9e-kube-api-access-rc5gm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.369197 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.369094    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:36.503593 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:36.503569    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 29 11:24:36.505754 ip-10-0-140-240 kubenswrapper[2584]: W0729 11:24:36.505726    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926c386f_291b_4d77_b823_38865ee5fc9e.slice/crio-5969f03d9add3561b6fd59f96331be01a1f873ab660f727f59a9aa98a417f733 WatchSource:0}: Error finding container 5969f03d9add3561b6fd59f96331be01a1f873ab660f727f59a9aa98a417f733: Status 404 returned error can't find the container with id 5969f03d9add3561b6fd59f96331be01a1f873ab660f727f59a9aa98a417f733
Jul 29 11:24:37.190153 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:37.190121    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"926c386f-291b-4d77-b823-38865ee5fc9e","Type":"ContainerStarted","Data":"2296588d4c034a0ea622d373d2199e70915ce8765b19dc9e321ff13ef156a452"}
Jul 29 11:24:37.190565 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:37.190172    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"926c386f-291b-4d77-b823-38865ee5fc9e","Type":"ContainerStarted","Data":"5969f03d9add3561b6fd59f96331be01a1f873ab660f727f59a9aa98a417f733"}
Jul 29 11:24:37.208344 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:37.208289    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podStartSLOduration=1.208273068 podStartE2EDuration="1.208273068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:24:37.207443508 +0000 UTC m=+3345.400470855" watchObservedRunningTime="2026-07-29 11:24:37.208273068 +0000 UTC m=+3345.401300415"
Jul 29 11:24:37.370186 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:37.370150    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:46.370170 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:46.370134    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:46.382971 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:46.382945    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:24:47.248414 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:24:47.248384    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:27:02.554478 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:02.554447    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schexzxnd_701ba11c-e9ed-4ead-9b4f-39ec15efcd62/main/0.log"
Jul 29 11:27:07.296310 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:07.296272    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"]
Jul 29 11:27:07.299441 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:07.299424    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"
Jul 29 11:27:07.302311 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:07.302289    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 29 11:27:07.302422 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:07.302297    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 29 11:27:07.303645 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:07.303629    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-5lcx4\""
Jul 29 11:27:07.312173 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:07.312153    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"]
Jul 29 11:27:07.400615 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:07.400580    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6mz9\" (UniqueName: \"kubernetes.io/projected/7a990c27-3285-4979-ab48-305eb3269d72-kube-api-access-d6mz9\") pod \"e2e-pvc-model-download-zst94\" (UID: \"7a990c27-3285-4979-ab48-305eb3269d72\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"
Jul 29 11:27:07.501203 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:07.501165    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6mz9\" (UniqueName: \"kubernetes.io/projected/7a990c27-3285-4979-ab48-305eb3269d72-kube-api-access-d6mz9\") pod \"e2e-pvc-model-download-zst94\" (UID: \"7a990c27-3285-4979-ab48-305eb3269d72\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"
Jul 29 11:27:07.509265 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:07.509242    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6mz9\" (UniqueName: \"kubernetes.io/projected/7a990c27-3285-4979-ab48-305eb3269d72-kube-api-access-d6mz9\") pod \"e2e-pvc-model-download-zst94\" (UID: \"7a990c27-3285-4979-ab48-305eb3269d72\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"
Jul 29 11:27:11.033319 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:11.033047    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") pod \"e2e-pvc-model-download-zst94\" (UID: \"7a990c27-3285-4979-ab48-305eb3269d72\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"
Jul 29 11:27:11.041517 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:11.041489    2584 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") pod \"e2e-pvc-model-download-zst94\" (UID: \"7a990c27-3285-4979-ab48-305eb3269d72\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"
Jul 29 11:27:11.133960 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:11.133922    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") pod \"e2e-pvc-model-download-zst94\" (UID: \"7a990c27-3285-4979-ab48-305eb3269d72\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"
Jul 29 11:27:11.134148 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:11.134072    2584 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") pod \"e2e-pvc-model-download-zst94\" (UID: \"7a990c27-3285-4979-ab48-305eb3269d72\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"
Jul 29 11:27:11.137157 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:11.137141    2584 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") pod \"e2e-pvc-model-download-zst94\" (UID: \"7a990c27-3285-4979-ab48-305eb3269d72\") DevicePath \"csi-8909cb5cb7ae688d273824526f4088877fdc3363724b73764c1173a25238aa18\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"
Jul 29 11:27:11.351758 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:11.351725    2584 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") pod \"e2e-pvc-model-download-zst94\" (UID: \"7a990c27-3285-4979-ab48-305eb3269d72\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/82708568a91e8258bb13db9f7d1800f3b22eae7de45eb7c99da7ab5491d26471/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"
Jul 29 11:27:11.365227 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:11.365203    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") pod \"e2e-pvc-model-download-zst94\" (UID: \"7a990c27-3285-4979-ab48-305eb3269d72\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"
Jul 29 11:27:11.508224 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:11.508194    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"
Jul 29 11:27:11.635152 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:11.635122    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"]
Jul 29 11:27:11.637623 ip-10-0-140-240 kubenswrapper[2584]: W0729 11:27:11.637596    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a990c27_3285_4979_ab48_305eb3269d72.slice/crio-e541f61d330cf96bcb8da5dc205988376bf201dbf801ad42cf431ed8a5908fd6 WatchSource:0}: Error finding container e541f61d330cf96bcb8da5dc205988376bf201dbf801ad42cf431ed8a5908fd6: Status 404 returned error can't find the container with id e541f61d330cf96bcb8da5dc205988376bf201dbf801ad42cf431ed8a5908fd6
Jul 29 11:27:11.879853 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:11.879751    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94" event={"ID":"7a990c27-3285-4979-ab48-305eb3269d72","Type":"ContainerStarted","Data":"d64cc959baa5bc171a7106a1504181590d3010b9100bf3f8e30275a120886a79"}
Jul 29 11:27:11.879853 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:11.879805    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94" event={"ID":"7a990c27-3285-4979-ab48-305eb3269d72","Type":"ContainerStarted","Data":"e541f61d330cf96bcb8da5dc205988376bf201dbf801ad42cf431ed8a5908fd6"}
Jul 29 11:27:11.895908 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:11.895850    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94" podStartSLOduration=7.8958356 podStartE2EDuration="7.8958356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 11:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:27:11.893834125 +0000 UTC m=+3500.086861500" watchObservedRunningTime="2026-07-29 11:27:11.8958356 +0000 UTC m=+3500.088862947"
Jul 29 11:27:18.917490 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:18.917453    2584 generic.go:356] "Generic (PLEG): container finished" podID="7a990c27-3285-4979-ab48-305eb3269d72" containerID="d64cc959baa5bc171a7106a1504181590d3010b9100bf3f8e30275a120886a79" exitCode=0
Jul 29 11:27:18.918060 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:18.917536    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94" event={"ID":"7a990c27-3285-4979-ab48-305eb3269d72","Type":"ContainerDied","Data":"d64cc959baa5bc171a7106a1504181590d3010b9100bf3f8e30275a120886a79"}
Jul 29 11:27:20.083429 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:20.083406    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"
Jul 29 11:27:20.220357 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:20.220256    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6mz9\" (UniqueName: \"kubernetes.io/projected/7a990c27-3285-4979-ab48-305eb3269d72-kube-api-access-d6mz9\") pod \"7a990c27-3285-4979-ab48-305eb3269d72\" (UID: \"7a990c27-3285-4979-ab48-305eb3269d72\") "
Jul 29 11:27:20.220532 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:20.220467    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") pod \"7a990c27-3285-4979-ab48-305eb3269d72\" (UID: \"7a990c27-3285-4979-ab48-305eb3269d72\") "
Jul 29 11:27:20.222515 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:20.222486    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a990c27-3285-4979-ab48-305eb3269d72-kube-api-access-d6mz9" (OuterVolumeSpecName: "kube-api-access-d6mz9") pod "7a990c27-3285-4979-ab48-305eb3269d72" (UID: "7a990c27-3285-4979-ab48-305eb3269d72"). InnerVolumeSpecName "kube-api-access-d6mz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:27:20.224337 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:20.224318    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221" (OuterVolumeSpecName: "model-storage") pod "7a990c27-3285-4979-ab48-305eb3269d72" (UID: "7a990c27-3285-4979-ab48-305eb3269d72"). InnerVolumeSpecName "pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 29 11:27:20.321572 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:20.321536    2584 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") on node \"ip-10-0-140-240.ec2.internal\" "
Jul 29 11:27:20.321843 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:20.321821    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6mz9\" (UniqueName: \"kubernetes.io/projected/7a990c27-3285-4979-ab48-305eb3269d72-kube-api-access-d6mz9\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:27:20.429693 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:20.429657    2584 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221") on node "ip-10-0-140-240.ec2.internal"
Jul 29 11:27:20.523947 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:20.523857    2584 reconciler_common.go:299] "Volume detached for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"csi-8909cb5cb7ae688d273824526f4088877fdc3363724b73764c1173a25238aa18\""
Jul 29 11:27:20.927189 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:20.927158    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94"
Jul 29 11:27:20.927372 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:20.927203    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-zst94" event={"ID":"7a990c27-3285-4979-ab48-305eb3269d72","Type":"ContainerDied","Data":"e541f61d330cf96bcb8da5dc205988376bf201dbf801ad42cf431ed8a5908fd6"}
Jul 29 11:27:20.927372 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:27:20.927240    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e541f61d330cf96bcb8da5dc205988376bf201dbf801ad42cf431ed8a5908fd6"
Jul 29 11:28:06.746583 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.746552    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 29 11:28:06.747095 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.746894    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a990c27-3285-4979-ab48-305eb3269d72" containerName="storage-initializer"
Jul 29 11:28:06.747095 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.746907    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a990c27-3285-4979-ab48-305eb3269d72" containerName="storage-initializer"
Jul 29 11:28:06.747095 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.746982    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a990c27-3285-4979-ab48-305eb3269d72" containerName="storage-initializer"
Jul 29 11:28:06.750004 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.749982    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.753924 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.753898    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 29 11:28:06.753924 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.753918    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 29 11:28:06.754121 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.753959    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 29 11:28:06.754121 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.753966    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-5lcx4\""
Jul 29 11:28:06.754121 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.753900    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-brhl9\""
Jul 29 11:28:06.760878 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.760855    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 29 11:28:06.838860 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.838829    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c0efa00-4094-45d8-93e5-052c4139f31a-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.838860 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.838866    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c0efa00-4094-45d8-93e5-052c4139f31a-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.839087 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.838885    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c0efa00-4094-45d8-93e5-052c4139f31a-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.839087 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.838911    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c0efa00-4094-45d8-93e5-052c4139f31a-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.839087 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.838968    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c0efa00-4094-45d8-93e5-052c4139f31a-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.839087 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.839041    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t72bv\" (UniqueName: \"kubernetes.io/projected/0c0efa00-4094-45d8-93e5-052c4139f31a-kube-api-access-t72bv\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.940067 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.940029    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t72bv\" (UniqueName: \"kubernetes.io/projected/0c0efa00-4094-45d8-93e5-052c4139f31a-kube-api-access-t72bv\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.940255 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.940084    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c0efa00-4094-45d8-93e5-052c4139f31a-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.940255 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.940110    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c0efa00-4094-45d8-93e5-052c4139f31a-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.940255 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.940132    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c0efa00-4094-45d8-93e5-052c4139f31a-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.940255 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.940159    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c0efa00-4094-45d8-93e5-052c4139f31a-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.940255 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.940206    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c0efa00-4094-45d8-93e5-052c4139f31a-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.940551 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.940528    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c0efa00-4094-45d8-93e5-052c4139f31a-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.940600 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.940578    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c0efa00-4094-45d8-93e5-052c4139f31a-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.940637 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.940594    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c0efa00-4094-45d8-93e5-052c4139f31a-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.942376 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.942357    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c0efa00-4094-45d8-93e5-052c4139f31a-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.942609 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.942593    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c0efa00-4094-45d8-93e5-052c4139f31a-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:28:06.947869 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:28:06.947846    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t72bv\" (UniqueName: \"kubernetes.io/projected/0c0efa00-4094-45d8-93e5-052c4139f31a-kube-api-access-t72bv\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0c0efa00-4094-45d8-93e5-052c4139f31a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:30:09.760558 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:30:09.760509    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 11:30:10.712981 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:30:10.712948    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:32:13.714198 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:32:13.714151    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 11:32:14.283784 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:32:14.283749    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:34:17.285311 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:34:17.285269    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 11:34:17.862505 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:34:17.862474    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:36:20.863171 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:36:20.863076    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 11:36:21.421502 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:36:21.421471    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:38:24.422965 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:38:24.422924    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 11:38:24.982446 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:38:24.982409    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:39:54.664437 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:54.664405    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 29 11:39:54.675326 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:39:54.675286    2584 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 11:39:54.675326 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:39:54.675326 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 11:39:54.675326 ip-10-0-140-240 kubenswrapper[2584]:  > containerID="2296588d4c034a0ea622d373d2199e70915ce8765b19dc9e321ff13ef156a452" cmd=["/bin/sleep","15"]
Jul 29 11:39:54.675580 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:54.675336    2584 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" message="" err=<
Jul 29 11:39:54.675580 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:39:54.675580 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 11:39:54.675580 ip-10-0-140-240 kubenswrapper[2584]:  >
Jul 29 11:39:54.675580 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:39:54.675348    2584 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 11:39:54.675580 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:39:54.675580 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 11:39:54.675580 ip-10-0-140-240 kubenswrapper[2584]:  > pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podUID="926c386f-291b-4d77-b823-38865ee5fc9e" containerName="main" containerID="cri-o://2296588d4c034a0ea622d373d2199e70915ce8765b19dc9e321ff13ef156a452"
Jul 29 11:39:54.675580 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:54.675381    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podUID="926c386f-291b-4d77-b823-38865ee5fc9e" containerName="main" containerID="cri-o://2296588d4c034a0ea622d373d2199e70915ce8765b19dc9e321ff13ef156a452" gracePeriod=60
Jul 29 11:39:55.024503 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.024478    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:39:55.151166 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.151132    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-model-cache\") pod \"926c386f-291b-4d77-b823-38865ee5fc9e\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") "
Jul 29 11:39:55.151351 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.151184    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-dshm\") pod \"926c386f-291b-4d77-b823-38865ee5fc9e\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") "
Jul 29 11:39:55.151351 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.151202    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-tmp-dir\") pod \"926c386f-291b-4d77-b823-38865ee5fc9e\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") "
Jul 29 11:39:55.151351 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.151235    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-home\") pod \"926c386f-291b-4d77-b823-38865ee5fc9e\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") "
Jul 29 11:39:55.151351 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.151255    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc5gm\" (UniqueName: \"kubernetes.io/projected/926c386f-291b-4d77-b823-38865ee5fc9e-kube-api-access-rc5gm\") pod \"926c386f-291b-4d77-b823-38865ee5fc9e\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") "
Jul 29 11:39:55.151351 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.151304    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/926c386f-291b-4d77-b823-38865ee5fc9e-tls-certs\") pod \"926c386f-291b-4d77-b823-38865ee5fc9e\" (UID: \"926c386f-291b-4d77-b823-38865ee5fc9e\") "
Jul 29 11:39:55.151611 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.151455    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "926c386f-291b-4d77-b823-38865ee5fc9e" (UID: "926c386f-291b-4d77-b823-38865ee5fc9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:39:55.151611 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.151528    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "926c386f-291b-4d77-b823-38865ee5fc9e" (UID: "926c386f-291b-4d77-b823-38865ee5fc9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:39:55.151611 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.151542    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-home" (OuterVolumeSpecName: "home") pod "926c386f-291b-4d77-b823-38865ee5fc9e" (UID: "926c386f-291b-4d77-b823-38865ee5fc9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:39:55.153412 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.153362    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-dshm" (OuterVolumeSpecName: "dshm") pod "926c386f-291b-4d77-b823-38865ee5fc9e" (UID: "926c386f-291b-4d77-b823-38865ee5fc9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:39:55.153517 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.153440    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926c386f-291b-4d77-b823-38865ee5fc9e-kube-api-access-rc5gm" (OuterVolumeSpecName: "kube-api-access-rc5gm") pod "926c386f-291b-4d77-b823-38865ee5fc9e" (UID: "926c386f-291b-4d77-b823-38865ee5fc9e"). InnerVolumeSpecName "kube-api-access-rc5gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:39:55.153517 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.153442    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926c386f-291b-4d77-b823-38865ee5fc9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "926c386f-291b-4d77-b823-38865ee5fc9e" (UID: "926c386f-291b-4d77-b823-38865ee5fc9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:39:55.252238 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.252153    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-dshm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:39:55.252238 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.252182    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:39:55.252238 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.252193    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-home\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:39:55.252238 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.252201    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc5gm\" (UniqueName: \"kubernetes.io/projected/926c386f-291b-4d77-b823-38865ee5fc9e-kube-api-access-rc5gm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:39:55.252238 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.252209    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/926c386f-291b-4d77-b823-38865ee5fc9e-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:39:55.252238 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.252219    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/926c386f-291b-4d77-b823-38865ee5fc9e-model-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:39:55.390549 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.390506    2584 generic.go:356] "Generic (PLEG): container finished" podID="926c386f-291b-4d77-b823-38865ee5fc9e" containerID="2296588d4c034a0ea622d373d2199e70915ce8765b19dc9e321ff13ef156a452" exitCode=0
Jul 29 11:39:55.390738 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.390555    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 29 11:39:55.390738 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.390596    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"926c386f-291b-4d77-b823-38865ee5fc9e","Type":"ContainerDied","Data":"2296588d4c034a0ea622d373d2199e70915ce8765b19dc9e321ff13ef156a452"}
Jul 29 11:39:55.390738 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.390652    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"926c386f-291b-4d77-b823-38865ee5fc9e","Type":"ContainerDied","Data":"5969f03d9add3561b6fd59f96331be01a1f873ab660f727f59a9aa98a417f733"}
Jul 29 11:39:55.390738 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.390697    2584 scope.go:117] "RemoveContainer" containerID="2296588d4c034a0ea622d373d2199e70915ce8765b19dc9e321ff13ef156a452"
Jul 29 11:39:55.400354 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.400338    2584 scope.go:117] "RemoveContainer" containerID="2296588d4c034a0ea622d373d2199e70915ce8765b19dc9e321ff13ef156a452"
Jul 29 11:39:55.400580 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:39:55.400558    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2296588d4c034a0ea622d373d2199e70915ce8765b19dc9e321ff13ef156a452\": container with ID starting with 2296588d4c034a0ea622d373d2199e70915ce8765b19dc9e321ff13ef156a452 not found: ID does not exist" containerID="2296588d4c034a0ea622d373d2199e70915ce8765b19dc9e321ff13ef156a452"
Jul 29 11:39:55.400645 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.400592    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2296588d4c034a0ea622d373d2199e70915ce8765b19dc9e321ff13ef156a452"} err="failed to get container status \"2296588d4c034a0ea622d373d2199e70915ce8765b19dc9e321ff13ef156a452\": rpc error: code = NotFound desc = could not find container \"2296588d4c034a0ea622d373d2199e70915ce8765b19dc9e321ff13ef156a452\": container with ID starting with 2296588d4c034a0ea622d373d2199e70915ce8765b19dc9e321ff13ef156a452 not found: ID does not exist"
Jul 29 11:39:55.411316 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.411292    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 29 11:39:55.415263 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:55.415235    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 29 11:39:56.257443 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:39:56.257411    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="926c386f-291b-4d77-b823-38865ee5fc9e" path="/var/lib/kubelet/pods/926c386f-291b-4d77-b823-38865ee5fc9e/volumes"
Jul 29 11:40:27.982983 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:40:27.982942    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 11:40:28.540757 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:28.540722    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:40:33.921779 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.921746    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 29 11:40:33.922252 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.922081    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="926c386f-291b-4d77-b823-38865ee5fc9e" containerName="main"
Jul 29 11:40:33.922252 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.922092    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="926c386f-291b-4d77-b823-38865ee5fc9e" containerName="main"
Jul 29 11:40:33.922252 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.922145    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="926c386f-291b-4d77-b823-38865ee5fc9e" containerName="main"
Jul 29 11:40:33.925029 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.925012    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:33.929195 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.929135    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-gp96d\""
Jul 29 11:40:33.929195 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.929181    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-mn-dockercfg-j74zl\""
Jul 29 11:40:33.929400 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.929197    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\""
Jul 29 11:40:33.929400 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.929222    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\""
Jul 29 11:40:33.929400 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.929333    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\""
Jul 29 11:40:33.934813 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.934792    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 29 11:40:33.997374 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.997346    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-home\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:33.997374 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.997377    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:33.997635 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.997489    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5ba817-13a0-42fc-9a33-95413ae07424-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:33.997635 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.997523    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xv6c\" (UniqueName: \"kubernetes.io/projected/4f5ba817-13a0-42fc-9a33-95413ae07424-kube-api-access-4xv6c\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:33.997635 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.997550    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:33.997635 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:33.997616    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-dshm\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:34.098665 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.098631    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5ba817-13a0-42fc-9a33-95413ae07424-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:34.098863 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.098699    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xv6c\" (UniqueName: \"kubernetes.io/projected/4f5ba817-13a0-42fc-9a33-95413ae07424-kube-api-access-4xv6c\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:34.098863 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.098719    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:34.098863 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.098751    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-dshm\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:34.098863 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.098776    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-home\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:34.099087 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.099041    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:34.099202 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.099182    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:34.099269 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.099207    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-home\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:34.099306 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.099291    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:34.100903 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.100886    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-dshm\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:34.101013 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.100998    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5ba817-13a0-42fc-9a33-95413ae07424-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:34.105987 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.105962    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xv6c\" (UniqueName: \"kubernetes.io/projected/4f5ba817-13a0-42fc-9a33-95413ae07424-kube-api-access-4xv6c\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:34.235862 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.235770    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:34.371158 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.371129    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 29 11:40:34.373690 ip-10-0-140-240 kubenswrapper[2584]: W0729 11:40:34.373647    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5ba817_13a0_42fc_9a33_95413ae07424.slice/crio-f2610f6127a37c25d2ad32fed1c4878929d379122188b8127552a33cc67e510a WatchSource:0}: Error finding container f2610f6127a37c25d2ad32fed1c4878929d379122188b8127552a33cc67e510a: Status 404 returned error can't find the container with id f2610f6127a37c25d2ad32fed1c4878929d379122188b8127552a33cc67e510a
Jul 29 11:40:34.569421 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.569389    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"4f5ba817-13a0-42fc-9a33-95413ae07424","Type":"ContainerStarted","Data":"14d794e71539463bc7b89d9920fa313681e0a2d17d75a99d3a374b82a0f74147"}
Jul 29 11:40:34.569618 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.569443    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"4f5ba817-13a0-42fc-9a33-95413ae07424","Type":"ContainerStarted","Data":"f2610f6127a37c25d2ad32fed1c4878929d379122188b8127552a33cc67e510a"}
Jul 29 11:40:34.587897 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:34.587847    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podStartSLOduration=1.587830355 podStartE2EDuration="1.587830355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:40:34.586221621 +0000 UTC m=+4302.779248979" watchObservedRunningTime="2026-07-29 11:40:34.587830355 +0000 UTC m=+4302.780857702"
Jul 29 11:40:35.235935 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:35.235903    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:44.237004 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:44.236923    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:44.255283 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:44.255257    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:40:44.628551 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:40:44.628523    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:42:31.542244 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:42:31.542159    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 11:42:32.106491 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:42:32.106453    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:43:39.779418 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:39.779381    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"]
Jul 29 11:43:39.782782 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:39.782762    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"
Jul 29 11:43:39.785818 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:39.785794    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 29 11:43:39.785967 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:39.785827    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 29 11:43:39.785967 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:39.785796    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 29 11:43:39.785967 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:39.785915    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-59zpd\""
Jul 29 11:43:39.786103 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:39.785970    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-cvv9k\""
Jul 29 11:43:39.790885 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:39.790864    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"]
Jul 29 11:43:39.862075 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:39.862044    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d2a082b-c43a-4097-a1e4-9d1a6d001699-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79\" (UID: \"7d2a082b-c43a-4097-a1e4-9d1a6d001699\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"
Jul 29 11:43:39.862262 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:39.862093    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm82x\" (UniqueName: \"kubernetes.io/projected/7d2a082b-c43a-4097-a1e4-9d1a6d001699-kube-api-access-zm82x\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79\" (UID: \"7d2a082b-c43a-4097-a1e4-9d1a6d001699\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"
Jul 29 11:43:39.963227 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:39.963194    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d2a082b-c43a-4097-a1e4-9d1a6d001699-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79\" (UID: \"7d2a082b-c43a-4097-a1e4-9d1a6d001699\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"
Jul 29 11:43:39.963404 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:39.963244    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm82x\" (UniqueName: \"kubernetes.io/projected/7d2a082b-c43a-4097-a1e4-9d1a6d001699-kube-api-access-zm82x\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79\" (UID: \"7d2a082b-c43a-4097-a1e4-9d1a6d001699\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"
Jul 29 11:43:39.965692 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:39.965657    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d2a082b-c43a-4097-a1e4-9d1a6d001699-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79\" (UID: \"7d2a082b-c43a-4097-a1e4-9d1a6d001699\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"
Jul 29 11:43:39.971289 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:39.971265    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm82x\" (UniqueName: \"kubernetes.io/projected/7d2a082b-c43a-4097-a1e4-9d1a6d001699-kube-api-access-zm82x\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79\" (UID: \"7d2a082b-c43a-4097-a1e4-9d1a6d001699\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"
Jul 29 11:43:40.093774 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:40.093737    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"
Jul 29 11:43:40.221972 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:40.221948    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"]
Jul 29 11:43:40.224089 ip-10-0-140-240 kubenswrapper[2584]: W0729 11:43:40.224060    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2a082b_c43a_4097_a1e4_9d1a6d001699.slice/crio-0e82ecbd06db55dc85e9a559de1f5d3ca420cb431f079fbe1df1838bb8186b97 WatchSource:0}: Error finding container 0e82ecbd06db55dc85e9a559de1f5d3ca420cb431f079fbe1df1838bb8186b97: Status 404 returned error can't find the container with id 0e82ecbd06db55dc85e9a559de1f5d3ca420cb431f079fbe1df1838bb8186b97
Jul 29 11:43:40.425644 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:40.425568    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79" event={"ID":"7d2a082b-c43a-4097-a1e4-9d1a6d001699","Type":"ContainerStarted","Data":"ba50edfca9cd3ef85f1e89614b3eacbf45c643c6552bc4a279af862e82c17ba5"}
Jul 29 11:43:40.425644 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:40.425618    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79" event={"ID":"7d2a082b-c43a-4097-a1e4-9d1a6d001699","Type":"ContainerStarted","Data":"0e82ecbd06db55dc85e9a559de1f5d3ca420cb431f079fbe1df1838bb8186b97"}
Jul 29 11:43:40.425849 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:40.425706    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"
Jul 29 11:43:40.442545 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:43:40.442492    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79" podStartSLOduration=1.44247863 podStartE2EDuration="1.44247863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:43:40.441514757 +0000 UTC m=+4488.634542117" watchObservedRunningTime="2026-07-29 11:43:40.44247863 +0000 UTC m=+4488.635505978"
Jul 29 11:44:11.431833 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:44:11.431804    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"
Jul 29 11:44:35.107740 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:44:35.107671    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 11:44:35.701374 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:44:35.701345    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:46:06.089440 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:06.089400    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"]
Jul 29 11:46:11.432780 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:11.432733    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79" podUID="7d2a082b-c43a-4097-a1e4-9d1a6d001699" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:46:21.109382 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:21.109316    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79" podUID="7d2a082b-c43a-4097-a1e4-9d1a6d001699" containerName="main" containerID="cri-o://ba50edfca9cd3ef85f1e89614b3eacbf45c643c6552bc4a279af862e82c17ba5" gracePeriod=45
Jul 29 11:46:21.366812 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:21.366749    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"
Jul 29 11:46:21.444992 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:21.444962    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm82x\" (UniqueName: \"kubernetes.io/projected/7d2a082b-c43a-4097-a1e4-9d1a6d001699-kube-api-access-zm82x\") pod \"7d2a082b-c43a-4097-a1e4-9d1a6d001699\" (UID: \"7d2a082b-c43a-4097-a1e4-9d1a6d001699\") "
Jul 29 11:46:21.445170 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:21.445015    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d2a082b-c43a-4097-a1e4-9d1a6d001699-tls-certs\") pod \"7d2a082b-c43a-4097-a1e4-9d1a6d001699\" (UID: \"7d2a082b-c43a-4097-a1e4-9d1a6d001699\") "
Jul 29 11:46:21.447202 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:21.447173    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2a082b-c43a-4097-a1e4-9d1a6d001699-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d2a082b-c43a-4097-a1e4-9d1a6d001699" (UID: "7d2a082b-c43a-4097-a1e4-9d1a6d001699"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:46:21.447342 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:21.447228    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2a082b-c43a-4097-a1e4-9d1a6d001699-kube-api-access-zm82x" (OuterVolumeSpecName: "kube-api-access-zm82x") pod "7d2a082b-c43a-4097-a1e4-9d1a6d001699" (UID: "7d2a082b-c43a-4097-a1e4-9d1a6d001699"). InnerVolumeSpecName "kube-api-access-zm82x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:46:21.545959 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:21.545923    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm82x\" (UniqueName: \"kubernetes.io/projected/7d2a082b-c43a-4097-a1e4-9d1a6d001699-kube-api-access-zm82x\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:46:21.545959 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:21.545955    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d2a082b-c43a-4097-a1e4-9d1a6d001699-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:46:22.212625 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.212588    2584 generic.go:356] "Generic (PLEG): container finished" podID="7d2a082b-c43a-4097-a1e4-9d1a6d001699" containerID="ba50edfca9cd3ef85f1e89614b3eacbf45c643c6552bc4a279af862e82c17ba5" exitCode=0
Jul 29 11:46:22.213147 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.212643    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79" event={"ID":"7d2a082b-c43a-4097-a1e4-9d1a6d001699","Type":"ContainerDied","Data":"ba50edfca9cd3ef85f1e89614b3eacbf45c643c6552bc4a279af862e82c17ba5"}
Jul 29 11:46:22.213147 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.212646    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"
Jul 29 11:46:22.213147 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.212722    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79" event={"ID":"7d2a082b-c43a-4097-a1e4-9d1a6d001699","Type":"ContainerDied","Data":"0e82ecbd06db55dc85e9a559de1f5d3ca420cb431f079fbe1df1838bb8186b97"}
Jul 29 11:46:22.213147 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.212748    2584 scope.go:117] "RemoveContainer" containerID="ba50edfca9cd3ef85f1e89614b3eacbf45c643c6552bc4a279af862e82c17ba5"
Jul 29 11:46:22.221729 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.221705    2584 scope.go:117] "RemoveContainer" containerID="ba50edfca9cd3ef85f1e89614b3eacbf45c643c6552bc4a279af862e82c17ba5"
Jul 29 11:46:22.222038 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:46:22.222019    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba50edfca9cd3ef85f1e89614b3eacbf45c643c6552bc4a279af862e82c17ba5\": container with ID starting with ba50edfca9cd3ef85f1e89614b3eacbf45c643c6552bc4a279af862e82c17ba5 not found: ID does not exist" containerID="ba50edfca9cd3ef85f1e89614b3eacbf45c643c6552bc4a279af862e82c17ba5"
Jul 29 11:46:22.222096 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.222048    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba50edfca9cd3ef85f1e89614b3eacbf45c643c6552bc4a279af862e82c17ba5"} err="failed to get container status \"ba50edfca9cd3ef85f1e89614b3eacbf45c643c6552bc4a279af862e82c17ba5\": rpc error: code = NotFound desc = could not find container \"ba50edfca9cd3ef85f1e89614b3eacbf45c643c6552bc4a279af862e82c17ba5\": container with ID starting with ba50edfca9cd3ef85f1e89614b3eacbf45c643c6552bc4a279af862e82c17ba5 not found: ID does not exist"
Jul 29 11:46:22.234105 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.234083    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"]
Jul 29 11:46:22.239328 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.239304    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-hrz79"]
Jul 29 11:46:22.255776 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.255751    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d2a082b-c43a-4097-a1e4-9d1a6d001699" path="/var/lib/kubelet/pods/7d2a082b-c43a-4097-a1e4-9d1a6d001699/volumes"
Jul 29 11:46:22.770320 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.770293    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"]
Jul 29 11:46:22.770641 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.770628    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d2a082b-c43a-4097-a1e4-9d1a6d001699" containerName="main"
Jul 29 11:46:22.770728 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.770642    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d2a082b-c43a-4097-a1e4-9d1a6d001699" containerName="main"
Jul 29 11:46:22.770728 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.770725    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d2a082b-c43a-4097-a1e4-9d1a6d001699" containerName="main"
Jul 29 11:46:22.774926 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.774906    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"
Jul 29 11:46:22.777958 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.777861    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 29 11:46:22.777958 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.777902    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 29 11:46:22.777958 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.777915    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 29 11:46:22.777958 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.777930    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-cvv9k\""
Jul 29 11:46:22.777958 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.777930    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-xcn6p\""
Jul 29 11:46:22.782700 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.782662    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"]
Jul 29 11:46:22.856588 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.856556    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/906a299f-e0ae-483a-8554-5cfc5f62799a-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm\" (UID: \"906a299f-e0ae-483a-8554-5cfc5f62799a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"
Jul 29 11:46:22.856588 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.856590    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhnt5\" (UniqueName: \"kubernetes.io/projected/906a299f-e0ae-483a-8554-5cfc5f62799a-kube-api-access-xhnt5\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm\" (UID: \"906a299f-e0ae-483a-8554-5cfc5f62799a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"
Jul 29 11:46:22.957578 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.957541    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/906a299f-e0ae-483a-8554-5cfc5f62799a-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm\" (UID: \"906a299f-e0ae-483a-8554-5cfc5f62799a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"
Jul 29 11:46:22.957578 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.957578    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhnt5\" (UniqueName: \"kubernetes.io/projected/906a299f-e0ae-483a-8554-5cfc5f62799a-kube-api-access-xhnt5\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm\" (UID: \"906a299f-e0ae-483a-8554-5cfc5f62799a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"
Jul 29 11:46:22.960166 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.960141    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/906a299f-e0ae-483a-8554-5cfc5f62799a-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm\" (UID: \"906a299f-e0ae-483a-8554-5cfc5f62799a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"
Jul 29 11:46:22.965815 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:22.965782    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhnt5\" (UniqueName: \"kubernetes.io/projected/906a299f-e0ae-483a-8554-5cfc5f62799a-kube-api-access-xhnt5\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm\" (UID: \"906a299f-e0ae-483a-8554-5cfc5f62799a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"
Jul 29 11:46:23.086428 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:23.086393    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"
Jul 29 11:46:23.217156 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:23.217132    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"]
Jul 29 11:46:23.219458 ip-10-0-140-240 kubenswrapper[2584]: W0729 11:46:23.219433    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906a299f_e0ae_483a_8554_5cfc5f62799a.slice/crio-b274dc2cb57a6d4b01fdb92d4fea8016082677d88de594260314072a5085e452 WatchSource:0}: Error finding container b274dc2cb57a6d4b01fdb92d4fea8016082677d88de594260314072a5085e452: Status 404 returned error can't find the container with id b274dc2cb57a6d4b01fdb92d4fea8016082677d88de594260314072a5085e452
Jul 29 11:46:24.224556 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:24.224523    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm" event={"ID":"906a299f-e0ae-483a-8554-5cfc5f62799a","Type":"ContainerStarted","Data":"fd5dbd8cb0082ca956e709413cf47570adbf0809bdca797b79d9ad4c6f8b782e"}
Jul 29 11:46:24.225016 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:24.224608    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm" event={"ID":"906a299f-e0ae-483a-8554-5cfc5f62799a","Type":"ContainerStarted","Data":"b274dc2cb57a6d4b01fdb92d4fea8016082677d88de594260314072a5085e452"}
Jul 29 11:46:24.225016 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:24.224631    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"
Jul 29 11:46:24.241523 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:24.241474    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm" podStartSLOduration=2.241462008 podStartE2EDuration="2.241462008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:46:24.239998698 +0000 UTC m=+4652.433026048" watchObservedRunningTime="2026-07-29 11:46:24.241462008 +0000 UTC m=+4652.434489354"
Jul 29 11:46:38.702099 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:46:38.702054    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 11:46:39.295369 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:39.295338    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:46:55.230950 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:46:55.230918    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"
Jul 29 11:48:42.296090 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:48:42.296006    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 11:48:42.849131 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:48:42.849100    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:49:22.394996 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:22.394958    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"]
Jul 29 11:49:25.230395 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:25.230357    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm" podUID="906a299f-e0ae-483a-8554-5cfc5f62799a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:49:35.230931 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:35.230894    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm" podUID="906a299f-e0ae-483a-8554-5cfc5f62799a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:49:37.414669 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:37.414608    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm" podUID="906a299f-e0ae-483a-8554-5cfc5f62799a" containerName="main" containerID="cri-o://fd5dbd8cb0082ca956e709413cf47570adbf0809bdca797b79d9ad4c6f8b782e" gracePeriod=45
Jul 29 11:49:37.661188 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:37.661162    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"
Jul 29 11:49:37.755562 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:37.755478    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhnt5\" (UniqueName: \"kubernetes.io/projected/906a299f-e0ae-483a-8554-5cfc5f62799a-kube-api-access-xhnt5\") pod \"906a299f-e0ae-483a-8554-5cfc5f62799a\" (UID: \"906a299f-e0ae-483a-8554-5cfc5f62799a\") "
Jul 29 11:49:37.755562 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:37.755513    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/906a299f-e0ae-483a-8554-5cfc5f62799a-tls-certs\") pod \"906a299f-e0ae-483a-8554-5cfc5f62799a\" (UID: \"906a299f-e0ae-483a-8554-5cfc5f62799a\") "
Jul 29 11:49:37.757544 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:37.757521    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/906a299f-e0ae-483a-8554-5cfc5f62799a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "906a299f-e0ae-483a-8554-5cfc5f62799a" (UID: "906a299f-e0ae-483a-8554-5cfc5f62799a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:49:37.757660 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:37.757637    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906a299f-e0ae-483a-8554-5cfc5f62799a-kube-api-access-xhnt5" (OuterVolumeSpecName: "kube-api-access-xhnt5") pod "906a299f-e0ae-483a-8554-5cfc5f62799a" (UID: "906a299f-e0ae-483a-8554-5cfc5f62799a"). InnerVolumeSpecName "kube-api-access-xhnt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:49:37.856112 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:37.856076    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhnt5\" (UniqueName: \"kubernetes.io/projected/906a299f-e0ae-483a-8554-5cfc5f62799a-kube-api-access-xhnt5\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:49:37.856112 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:37.856107    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/906a299f-e0ae-483a-8554-5cfc5f62799a-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:49:38.114382 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:38.114342    2584 generic.go:356] "Generic (PLEG): container finished" podID="906a299f-e0ae-483a-8554-5cfc5f62799a" containerID="fd5dbd8cb0082ca956e709413cf47570adbf0809bdca797b79d9ad4c6f8b782e" exitCode=0
Jul 29 11:49:38.114382 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:38.114387    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"
Jul 29 11:49:38.114627 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:38.114423    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm" event={"ID":"906a299f-e0ae-483a-8554-5cfc5f62799a","Type":"ContainerDied","Data":"fd5dbd8cb0082ca956e709413cf47570adbf0809bdca797b79d9ad4c6f8b782e"}
Jul 29 11:49:38.114627 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:38.114474    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm" event={"ID":"906a299f-e0ae-483a-8554-5cfc5f62799a","Type":"ContainerDied","Data":"b274dc2cb57a6d4b01fdb92d4fea8016082677d88de594260314072a5085e452"}
Jul 29 11:49:38.114627 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:38.114503    2584 scope.go:117] "RemoveContainer" containerID="fd5dbd8cb0082ca956e709413cf47570adbf0809bdca797b79d9ad4c6f8b782e"
Jul 29 11:49:38.125875 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:38.125851    2584 scope.go:117] "RemoveContainer" containerID="fd5dbd8cb0082ca956e709413cf47570adbf0809bdca797b79d9ad4c6f8b782e"
Jul 29 11:49:38.126235 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:49:38.126195    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd5dbd8cb0082ca956e709413cf47570adbf0809bdca797b79d9ad4c6f8b782e\": container with ID starting with fd5dbd8cb0082ca956e709413cf47570adbf0809bdca797b79d9ad4c6f8b782e not found: ID does not exist" containerID="fd5dbd8cb0082ca956e709413cf47570adbf0809bdca797b79d9ad4c6f8b782e"
Jul 29 11:49:38.126235 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:38.126228    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd5dbd8cb0082ca956e709413cf47570adbf0809bdca797b79d9ad4c6f8b782e"} err="failed to get container status \"fd5dbd8cb0082ca956e709413cf47570adbf0809bdca797b79d9ad4c6f8b782e\": rpc error: code = NotFound desc = could not find container \"fd5dbd8cb0082ca956e709413cf47570adbf0809bdca797b79d9ad4c6f8b782e\": container with ID starting with fd5dbd8cb0082ca956e709413cf47570adbf0809bdca797b79d9ad4c6f8b782e not found: ID does not exist"
Jul 29 11:49:38.135751 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:38.135728    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"]
Jul 29 11:49:38.138153 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:38.138132    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-68lhm"]
Jul 29 11:49:38.255333 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:49:38.255304    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="906a299f-e0ae-483a-8554-5cfc5f62799a" path="/var/lib/kubelet/pods/906a299f-e0ae-483a-8554-5cfc5f62799a/volumes"
Jul 29 11:50:45.849927 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:50:45.849882    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 11:50:46.431488 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:50:46.431455    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:52:49.432855 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:52:49.432812    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 11:52:50.019664 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:52:50.019634    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:54:53.020523 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:54:53.020479    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 11:54:53.603109 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:54:53.603077    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:55:24.061309 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.061276    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 29 11:55:24.070170 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:55:24.070135    2584 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 11:55:24.070170 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:55:24.070170 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 11:55:24.070170 ip-10-0-140-240 kubenswrapper[2584]:  > containerID="14d794e71539463bc7b89d9920fa313681e0a2d17d75a99d3a374b82a0f74147" cmd=["/bin/sleep","15"]
Jul 29 11:55:24.070498 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.070184    2584 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" message="" err=<
Jul 29 11:55:24.070498 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:55:24.070498 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 11:55:24.070498 ip-10-0-140-240 kubenswrapper[2584]:  >
Jul 29 11:55:24.070498 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:55:24.070197    2584 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 11:55:24.070498 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:55:24.070498 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 11:55:24.070498 ip-10-0-140-240 kubenswrapper[2584]:  > pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podUID="4f5ba817-13a0-42fc-9a33-95413ae07424" containerName="main" containerID="cri-o://14d794e71539463bc7b89d9920fa313681e0a2d17d75a99d3a374b82a0f74147"
Jul 29 11:55:24.070498 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.070242    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podUID="4f5ba817-13a0-42fc-9a33-95413ae07424" containerName="main" containerID="cri-o://14d794e71539463bc7b89d9920fa313681e0a2d17d75a99d3a374b82a0f74147" gracePeriod=60
Jul 29 11:55:24.416190 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.416165    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:55:24.429194 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.429170    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-home\") pod \"4f5ba817-13a0-42fc-9a33-95413ae07424\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") "
Jul 29 11:55:24.429348 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.429209    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-tmp-dir\") pod \"4f5ba817-13a0-42fc-9a33-95413ae07424\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") "
Jul 29 11:55:24.429348 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.429234    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5ba817-13a0-42fc-9a33-95413ae07424-tls-certs\") pod \"4f5ba817-13a0-42fc-9a33-95413ae07424\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") "
Jul 29 11:55:24.429348 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.429280    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-dshm\") pod \"4f5ba817-13a0-42fc-9a33-95413ae07424\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") "
Jul 29 11:55:24.429348 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.429341    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xv6c\" (UniqueName: \"kubernetes.io/projected/4f5ba817-13a0-42fc-9a33-95413ae07424-kube-api-access-4xv6c\") pod \"4f5ba817-13a0-42fc-9a33-95413ae07424\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") "
Jul 29 11:55:24.429565 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.429387    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-model-cache\") pod \"4f5ba817-13a0-42fc-9a33-95413ae07424\" (UID: \"4f5ba817-13a0-42fc-9a33-95413ae07424\") "
Jul 29 11:55:24.429565 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.429467    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-home" (OuterVolumeSpecName: "home") pod "4f5ba817-13a0-42fc-9a33-95413ae07424" (UID: "4f5ba817-13a0-42fc-9a33-95413ae07424"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:55:24.429565 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.429473    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f5ba817-13a0-42fc-9a33-95413ae07424" (UID: "4f5ba817-13a0-42fc-9a33-95413ae07424"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:55:24.429750 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.429669    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f5ba817-13a0-42fc-9a33-95413ae07424" (UID: "4f5ba817-13a0-42fc-9a33-95413ae07424"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:55:24.429750 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.429743    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-home\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:55:24.429863 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.429760    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:55:24.431608 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.431581    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f5ba817-13a0-42fc-9a33-95413ae07424-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f5ba817-13a0-42fc-9a33-95413ae07424" (UID: "4f5ba817-13a0-42fc-9a33-95413ae07424"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:55:24.431725 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.431687    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-dshm" (OuterVolumeSpecName: "dshm") pod "4f5ba817-13a0-42fc-9a33-95413ae07424" (UID: "4f5ba817-13a0-42fc-9a33-95413ae07424"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:55:24.431725 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.431704    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5ba817-13a0-42fc-9a33-95413ae07424-kube-api-access-4xv6c" (OuterVolumeSpecName: "kube-api-access-4xv6c") pod "4f5ba817-13a0-42fc-9a33-95413ae07424" (UID: "4f5ba817-13a0-42fc-9a33-95413ae07424"). InnerVolumeSpecName "kube-api-access-4xv6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:55:24.530124 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.530080    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-model-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:55:24.530124 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.530117    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5ba817-13a0-42fc-9a33-95413ae07424-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:55:24.530124 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.530128    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5ba817-13a0-42fc-9a33-95413ae07424-dshm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:55:24.530371 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.530139    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xv6c\" (UniqueName: \"kubernetes.io/projected/4f5ba817-13a0-42fc-9a33-95413ae07424-kube-api-access-4xv6c\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 11:55:24.761446 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.761342    2584 generic.go:356] "Generic (PLEG): container finished" podID="4f5ba817-13a0-42fc-9a33-95413ae07424" containerID="14d794e71539463bc7b89d9920fa313681e0a2d17d75a99d3a374b82a0f74147" exitCode=0
Jul 29 11:55:24.761446 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.761396    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"4f5ba817-13a0-42fc-9a33-95413ae07424","Type":"ContainerDied","Data":"14d794e71539463bc7b89d9920fa313681e0a2d17d75a99d3a374b82a0f74147"}
Jul 29 11:55:24.761729 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.761459    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"4f5ba817-13a0-42fc-9a33-95413ae07424","Type":"ContainerDied","Data":"f2610f6127a37c25d2ad32fed1c4878929d379122188b8127552a33cc67e510a"}
Jul 29 11:55:24.761729 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.761402    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 29 11:55:24.761729 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.761490    2584 scope.go:117] "RemoveContainer" containerID="14d794e71539463bc7b89d9920fa313681e0a2d17d75a99d3a374b82a0f74147"
Jul 29 11:55:24.772441 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.772369    2584 scope.go:117] "RemoveContainer" containerID="14d794e71539463bc7b89d9920fa313681e0a2d17d75a99d3a374b82a0f74147"
Jul 29 11:55:24.772863 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:55:24.772698    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d794e71539463bc7b89d9920fa313681e0a2d17d75a99d3a374b82a0f74147\": container with ID starting with 14d794e71539463bc7b89d9920fa313681e0a2d17d75a99d3a374b82a0f74147 not found: ID does not exist" containerID="14d794e71539463bc7b89d9920fa313681e0a2d17d75a99d3a374b82a0f74147"
Jul 29 11:55:24.772863 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.772734    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d794e71539463bc7b89d9920fa313681e0a2d17d75a99d3a374b82a0f74147"} err="failed to get container status \"14d794e71539463bc7b89d9920fa313681e0a2d17d75a99d3a374b82a0f74147\": rpc error: code = NotFound desc = could not find container \"14d794e71539463bc7b89d9920fa313681e0a2d17d75a99d3a374b82a0f74147\": container with ID starting with 14d794e71539463bc7b89d9920fa313681e0a2d17d75a99d3a374b82a0f74147 not found: ID does not exist"
Jul 29 11:55:24.786667 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.786638    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 29 11:55:24.790303 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:24.790277    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 29 11:55:26.255033 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:26.254998    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f5ba817-13a0-42fc-9a33-95413ae07424" path="/var/lib/kubelet/pods/4f5ba817-13a0-42fc-9a33-95413ae07424/volumes"
Jul 29 11:55:57.028886 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.028853    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"]
Jul 29 11:55:57.029352 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.029190    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906a299f-e0ae-483a-8554-5cfc5f62799a" containerName="main"
Jul 29 11:55:57.029352 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.029201    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="906a299f-e0ae-483a-8554-5cfc5f62799a" containerName="main"
Jul 29 11:55:57.029352 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.029210    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f5ba817-13a0-42fc-9a33-95413ae07424" containerName="main"
Jul 29 11:55:57.029352 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.029215    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f5ba817-13a0-42fc-9a33-95413ae07424" containerName="main"
Jul 29 11:55:57.029352 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.029276    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f5ba817-13a0-42fc-9a33-95413ae07424" containerName="main"
Jul 29 11:55:57.029352 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.029287    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="906a299f-e0ae-483a-8554-5cfc5f62799a" containerName="main"
Jul 29 11:55:57.033237 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.033219    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.037543 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.037517    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"kube-root-ca.crt\""
Jul 29 11:55:57.037730 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.037520    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"default-dockercfg-jgw5g\""
Jul 29 11:55:57.037730 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.037581    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"openshift-service-ca.crt\""
Jul 29 11:55:57.037730 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.037520    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"autoscale-cleanup-hpa-kserve-self-signed-certs\""
Jul 29 11:55:57.040936 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.040913    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"]
Jul 29 11:55:57.122407 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.122374    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-home\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.122407 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.122414    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-model-cache\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.122649 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.122509    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f88da38-d6bd-41f4-9402-310428813647-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.122649 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.122568    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-tmp-dir\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.122649 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.122599    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmjp8\" (UniqueName: \"kubernetes.io/projected/4f88da38-d6bd-41f4-9402-310428813647-kube-api-access-tmjp8\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.122774 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.122652    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-dshm\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.220965 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.220927    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"]
Jul 29 11:55:57.223539 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.223503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f88da38-d6bd-41f4-9402-310428813647-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.223670 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.223560    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-tmp-dir\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.223670 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.223589    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmjp8\" (UniqueName: \"kubernetes.io/projected/4f88da38-d6bd-41f4-9402-310428813647-kube-api-access-tmjp8\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.223670 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.223649    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-dshm\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.223863 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.223714    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-home\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.223863 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.223741    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-model-cache\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.224101 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.224074    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-tmp-dir\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.224278 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.224255    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"
Jul 29 11:55:57.224397 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.224165    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-model-cache\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.224459 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.224118    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-home\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.226473 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.226451    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-dshm\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.226604 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.226474    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f88da38-d6bd-41f4-9402-310428813647-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.228949 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.228926    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"autoscale-cleanup-hpa-epp-sa-dockercfg-d58nz\""
Jul 29 11:55:57.235177 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.235154    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmjp8\" (UniqueName: \"kubernetes.io/projected/4f88da38-d6bd-41f4-9402-310428813647-kube-api-access-tmjp8\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.245333 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.245307    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"]
Jul 29 11:55:57.324263 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.324220    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05530424-c6ce-4729-a9fa-349d5173399c-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw\" (UID: \"05530424-c6ce-4729-a9fa-349d5173399c\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"
Jul 29 11:55:57.324464 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.324281    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7djmq\" (UniqueName: \"kubernetes.io/projected/05530424-c6ce-4729-a9fa-349d5173399c-kube-api-access-7djmq\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw\" (UID: \"05530424-c6ce-4729-a9fa-349d5173399c\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"
Jul 29 11:55:57.345204 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.344874    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:55:57.425222 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.425188    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05530424-c6ce-4729-a9fa-349d5173399c-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw\" (UID: \"05530424-c6ce-4729-a9fa-349d5173399c\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"
Jul 29 11:55:57.425390 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.425272    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djmq\" (UniqueName: \"kubernetes.io/projected/05530424-c6ce-4729-a9fa-349d5173399c-kube-api-access-7djmq\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw\" (UID: \"05530424-c6ce-4729-a9fa-349d5173399c\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"
Jul 29 11:55:57.429095 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.429071    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05530424-c6ce-4729-a9fa-349d5173399c-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw\" (UID: \"05530424-c6ce-4729-a9fa-349d5173399c\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"
Jul 29 11:55:57.439016 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.438985    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7djmq\" (UniqueName: \"kubernetes.io/projected/05530424-c6ce-4729-a9fa-349d5173399c-kube-api-access-7djmq\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw\" (UID: \"05530424-c6ce-4729-a9fa-349d5173399c\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"
Jul 29 11:55:57.485686 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.485645    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"]
Jul 29 11:55:57.488998 ip-10-0-140-240 kubenswrapper[2584]: W0729 11:55:57.488970    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f88da38_d6bd_41f4_9402_310428813647.slice/crio-d9db30c8cdbcf7c8eb4d74691447346fe2967fb9e10853dc975f56924c73c3f4 WatchSource:0}: Error finding container d9db30c8cdbcf7c8eb4d74691447346fe2967fb9e10853dc975f56924c73c3f4: Status 404 returned error can't find the container with id d9db30c8cdbcf7c8eb4d74691447346fe2967fb9e10853dc975f56924c73c3f4
Jul 29 11:55:57.550145 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.550120    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"
Jul 29 11:55:57.703398 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.703369    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"]
Jul 29 11:55:57.707354 ip-10-0-140-240 kubenswrapper[2584]: W0729 11:55:57.707323    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05530424_c6ce_4729_a9fa_349d5173399c.slice/crio-4ec2078c1af276a4eedaa37ca61a59d467d15bd3e690e0b7c9f5e239026eea68 WatchSource:0}: Error finding container 4ec2078c1af276a4eedaa37ca61a59d467d15bd3e690e0b7c9f5e239026eea68: Status 404 returned error can't find the container with id 4ec2078c1af276a4eedaa37ca61a59d467d15bd3e690e0b7c9f5e239026eea68
Jul 29 11:55:57.924506 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.924401    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw" event={"ID":"05530424-c6ce-4729-a9fa-349d5173399c","Type":"ContainerStarted","Data":"c53b3698b4c17e17666d9e9a3c8eda2e78c4d9f8a4e443c10f985cb4cabf239b"}
Jul 29 11:55:57.924706 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.924594    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"
Jul 29 11:55:57.924706 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.924630    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw" event={"ID":"05530424-c6ce-4729-a9fa-349d5173399c","Type":"ContainerStarted","Data":"4ec2078c1af276a4eedaa37ca61a59d467d15bd3e690e0b7c9f5e239026eea68"}
Jul 29 11:55:57.925919 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.925892    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz" event={"ID":"4f88da38-d6bd-41f4-9402-310428813647","Type":"ContainerStarted","Data":"f3b353d7ab31b1f0dec54ba956402d365bc09bd84f786531e36153d11854b0e8"}
Jul 29 11:55:57.926050 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.925934    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz" event={"ID":"4f88da38-d6bd-41f4-9402-310428813647","Type":"ContainerStarted","Data":"d9db30c8cdbcf7c8eb4d74691447346fe2967fb9e10853dc975f56924c73c3f4"}
Jul 29 11:55:57.942283 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.942238    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw" podStartSLOduration=0.942225364 podStartE2EDuration="942.225364ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:55:57.941873464 +0000 UTC m=+5226.134900812" watchObservedRunningTime="2026-07-29 11:55:57.942225364 +0000 UTC m=+5226.135252710"
Jul 29 11:55:57.958785 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:57.958743    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz" podStartSLOduration=0.9587272 podStartE2EDuration="958.7272ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:55:57.957185302 +0000 UTC m=+5226.150212649" watchObservedRunningTime="2026-07-29 11:55:57.9587272 +0000 UTC m=+5226.151754548"
Jul 29 11:55:58.345260 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:55:58.345211    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:56:07.345356 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:56:07.345311    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:56:07.364332 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:56:07.364302    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:56:07.982804 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:56:07.982773    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 11:56:28.932072 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:56:28.932042    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"
Jul 29 11:56:56.604030 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:56:56.603991    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 11:56:57.194649 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:56:57.194617    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 11:59:00.195806 ip-10-0-140-240 kubenswrapper[2584]: E0729 11:59:00.195762    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 11:59:00.754715 ip-10-0-140-240 kubenswrapper[2584]: I0729 11:59:00.754663    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 12:01:03.756132 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:01:03.756084    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 12:01:04.343359 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:01:04.343326    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 12:03:07.344100 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:03:07.344052    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 12:03:07.911343 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:03:07.911310    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 12:05:10.912408 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:05:10.912363    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 12:05:11.487089 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:05:11.487057    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 12:07:14.488408 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:07:14.488363    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 12:07:15.076895 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:07:15.076861    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 12:09:18.078246 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:09:18.078155    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 12:09:18.629233 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:09:18.629201    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 12:11:21.630215 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:11:21.630172    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 12:11:22.200508 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:22.200478    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 12:11:39.617245 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:39.617203    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"]
Jul 29 12:11:39.617928 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:39.617896    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"]
Jul 29 12:11:39.627232 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:11:39.627191    2584 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 12:11:39.627232 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 12:11:39.627232 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 12:11:39.627232 ip-10-0-140-240 kubenswrapper[2584]:  > containerID="f3b353d7ab31b1f0dec54ba956402d365bc09bd84f786531e36153d11854b0e8" cmd=["/bin/sleep","15"]
Jul 29 12:11:39.627534 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:39.627249    2584 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz" message="" err=<
Jul 29 12:11:39.627534 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 12:11:39.627534 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 12:11:39.627534 ip-10-0-140-240 kubenswrapper[2584]:  >
Jul 29 12:11:39.627534 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:11:39.627265    2584 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 12:11:39.627534 ip-10-0-140-240 kubenswrapper[2584]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 12:11:39.627534 ip-10-0-140-240 kubenswrapper[2584]:         , stdout: , stderr: , exit code -1
Jul 29 12:11:39.627534 ip-10-0-140-240 kubenswrapper[2584]:  > pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz" podUID="4f88da38-d6bd-41f4-9402-310428813647" containerName="main" containerID="cri-o://f3b353d7ab31b1f0dec54ba956402d365bc09bd84f786531e36153d11854b0e8"
Jul 29 12:11:39.627534 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:39.627320    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz" podUID="4f88da38-d6bd-41f4-9402-310428813647" containerName="main" containerID="cri-o://f3b353d7ab31b1f0dec54ba956402d365bc09bd84f786531e36153d11854b0e8" gracePeriod=60
Jul 29 12:11:39.978694 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:39.978649    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 12:11:40.081491 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.081450    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-model-cache\") pod \"4f88da38-d6bd-41f4-9402-310428813647\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") "
Jul 29 12:11:40.081491 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.081490    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-tmp-dir\") pod \"4f88da38-d6bd-41f4-9402-310428813647\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") "
Jul 29 12:11:40.081779 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.081543    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-dshm\") pod \"4f88da38-d6bd-41f4-9402-310428813647\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") "
Jul 29 12:11:40.081779 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.081568    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-home\") pod \"4f88da38-d6bd-41f4-9402-310428813647\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") "
Jul 29 12:11:40.081779 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.081588    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f88da38-d6bd-41f4-9402-310428813647-tls-certs\") pod \"4f88da38-d6bd-41f4-9402-310428813647\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") "
Jul 29 12:11:40.081779 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.081622    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmjp8\" (UniqueName: \"kubernetes.io/projected/4f88da38-d6bd-41f4-9402-310428813647-kube-api-access-tmjp8\") pod \"4f88da38-d6bd-41f4-9402-310428813647\" (UID: \"4f88da38-d6bd-41f4-9402-310428813647\") "
Jul 29 12:11:40.081999 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.081830    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f88da38-d6bd-41f4-9402-310428813647" (UID: "4f88da38-d6bd-41f4-9402-310428813647"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:11:40.081999 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.081908    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-home" (OuterVolumeSpecName: "home") pod "4f88da38-d6bd-41f4-9402-310428813647" (UID: "4f88da38-d6bd-41f4-9402-310428813647"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:11:40.081999 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.081922    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f88da38-d6bd-41f4-9402-310428813647" (UID: "4f88da38-d6bd-41f4-9402-310428813647"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:11:40.083868 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.083847    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f88da38-d6bd-41f4-9402-310428813647-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f88da38-d6bd-41f4-9402-310428813647" (UID: "4f88da38-d6bd-41f4-9402-310428813647"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 12:11:40.083954 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.083859    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f88da38-d6bd-41f4-9402-310428813647-kube-api-access-tmjp8" (OuterVolumeSpecName: "kube-api-access-tmjp8") pod "4f88da38-d6bd-41f4-9402-310428813647" (UID: "4f88da38-d6bd-41f4-9402-310428813647"). InnerVolumeSpecName "kube-api-access-tmjp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 12:11:40.083954 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.083879    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-dshm" (OuterVolumeSpecName: "dshm") pod "4f88da38-d6bd-41f4-9402-310428813647" (UID: "4f88da38-d6bd-41f4-9402-310428813647"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:11:40.182861 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.182777    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmjp8\" (UniqueName: \"kubernetes.io/projected/4f88da38-d6bd-41f4-9402-310428813647-kube-api-access-tmjp8\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 12:11:40.182861 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.182819    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-model-cache\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 12:11:40.182861 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.182829    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-tmp-dir\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 12:11:40.182861 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.182839    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-dshm\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 12:11:40.182861 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.182847    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f88da38-d6bd-41f4-9402-310428813647-home\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 12:11:40.182861 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.182855    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f88da38-d6bd-41f4-9402-310428813647-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 12:11:40.282372 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.282338    2584 generic.go:356] "Generic (PLEG): container finished" podID="4f88da38-d6bd-41f4-9402-310428813647" containerID="f3b353d7ab31b1f0dec54ba956402d365bc09bd84f786531e36153d11854b0e8" exitCode=0
Jul 29 12:11:40.282518 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.282378    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"
Jul 29 12:11:40.282518 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.282426    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz" event={"ID":"4f88da38-d6bd-41f4-9402-310428813647","Type":"ContainerDied","Data":"f3b353d7ab31b1f0dec54ba956402d365bc09bd84f786531e36153d11854b0e8"}
Jul 29 12:11:40.282518 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.282475    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz" event={"ID":"4f88da38-d6bd-41f4-9402-310428813647","Type":"ContainerDied","Data":"d9db30c8cdbcf7c8eb4d74691447346fe2967fb9e10853dc975f56924c73c3f4"}
Jul 29 12:11:40.282518 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.282501    2584 scope.go:117] "RemoveContainer" containerID="f3b353d7ab31b1f0dec54ba956402d365bc09bd84f786531e36153d11854b0e8"
Jul 29 12:11:40.291413 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.291392    2584 scope.go:117] "RemoveContainer" containerID="f3b353d7ab31b1f0dec54ba956402d365bc09bd84f786531e36153d11854b0e8"
Jul 29 12:11:40.291666 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:11:40.291644    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b353d7ab31b1f0dec54ba956402d365bc09bd84f786531e36153d11854b0e8\": container with ID starting with f3b353d7ab31b1f0dec54ba956402d365bc09bd84f786531e36153d11854b0e8 not found: ID does not exist" containerID="f3b353d7ab31b1f0dec54ba956402d365bc09bd84f786531e36153d11854b0e8"
Jul 29 12:11:40.291736 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.291692    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b353d7ab31b1f0dec54ba956402d365bc09bd84f786531e36153d11854b0e8"} err="failed to get container status \"f3b353d7ab31b1f0dec54ba956402d365bc09bd84f786531e36153d11854b0e8\": rpc error: code = NotFound desc = could not find container \"f3b353d7ab31b1f0dec54ba956402d365bc09bd84f786531e36153d11854b0e8\": container with ID starting with f3b353d7ab31b1f0dec54ba956402d365bc09bd84f786531e36153d11854b0e8 not found: ID does not exist"
Jul 29 12:11:40.298388 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.298367    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"]
Jul 29 12:11:40.302374 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.302354    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz"]
Jul 29 12:11:40.973009 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:40.972955    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-8dkpz" podUID="4f88da38-d6bd-41f4-9402-310428813647" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 29 12:11:42.254669 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:42.254636    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f88da38-d6bd-41f4-9402-310428813647" path="/var/lib/kubelet/pods/4f88da38-d6bd-41f4-9402-310428813647/volumes"
Jul 29 12:11:48.932427 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:48.932385    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw" podUID="05530424-c6ce-4729-a9fa-349d5173399c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 12:11:54.645720 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:54.645653    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw" podUID="05530424-c6ce-4729-a9fa-349d5173399c" containerName="main" containerID="cri-o://c53b3698b4c17e17666d9e9a3c8eda2e78c4d9f8a4e443c10f985cb4cabf239b" gracePeriod=45
Jul 29 12:11:54.902387 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:54.902322    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"
Jul 29 12:11:55.012986 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:55.012936    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05530424-c6ce-4729-a9fa-349d5173399c-tls-certs\") pod \"05530424-c6ce-4729-a9fa-349d5173399c\" (UID: \"05530424-c6ce-4729-a9fa-349d5173399c\") "
Jul 29 12:11:55.012986 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:55.012993    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7djmq\" (UniqueName: \"kubernetes.io/projected/05530424-c6ce-4729-a9fa-349d5173399c-kube-api-access-7djmq\") pod \"05530424-c6ce-4729-a9fa-349d5173399c\" (UID: \"05530424-c6ce-4729-a9fa-349d5173399c\") "
Jul 29 12:11:55.015097 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:55.015067    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05530424-c6ce-4729-a9fa-349d5173399c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05530424-c6ce-4729-a9fa-349d5173399c" (UID: "05530424-c6ce-4729-a9fa-349d5173399c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 12:11:55.015156 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:55.015131    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05530424-c6ce-4729-a9fa-349d5173399c-kube-api-access-7djmq" (OuterVolumeSpecName: "kube-api-access-7djmq") pod "05530424-c6ce-4729-a9fa-349d5173399c" (UID: "05530424-c6ce-4729-a9fa-349d5173399c"). InnerVolumeSpecName "kube-api-access-7djmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 12:11:55.114573 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:55.114519    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05530424-c6ce-4729-a9fa-349d5173399c-tls-certs\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 12:11:55.114573 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:55.114566    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7djmq\" (UniqueName: \"kubernetes.io/projected/05530424-c6ce-4729-a9fa-349d5173399c-kube-api-access-7djmq\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 12:11:55.352964 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:55.352931    2584 generic.go:356] "Generic (PLEG): container finished" podID="05530424-c6ce-4729-a9fa-349d5173399c" containerID="c53b3698b4c17e17666d9e9a3c8eda2e78c4d9f8a4e443c10f985cb4cabf239b" exitCode=0
Jul 29 12:11:55.353150 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:55.352975    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"
Jul 29 12:11:55.353150 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:55.353012    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw" event={"ID":"05530424-c6ce-4729-a9fa-349d5173399c","Type":"ContainerDied","Data":"c53b3698b4c17e17666d9e9a3c8eda2e78c4d9f8a4e443c10f985cb4cabf239b"}
Jul 29 12:11:55.353150 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:55.353068    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw" event={"ID":"05530424-c6ce-4729-a9fa-349d5173399c","Type":"ContainerDied","Data":"4ec2078c1af276a4eedaa37ca61a59d467d15bd3e690e0b7c9f5e239026eea68"}
Jul 29 12:11:55.353150 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:55.353096    2584 scope.go:117] "RemoveContainer" containerID="c53b3698b4c17e17666d9e9a3c8eda2e78c4d9f8a4e443c10f985cb4cabf239b"
Jul 29 12:11:55.362360 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:55.362339    2584 scope.go:117] "RemoveContainer" containerID="c53b3698b4c17e17666d9e9a3c8eda2e78c4d9f8a4e443c10f985cb4cabf239b"
Jul 29 12:11:55.362625 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:11:55.362606    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c53b3698b4c17e17666d9e9a3c8eda2e78c4d9f8a4e443c10f985cb4cabf239b\": container with ID starting with c53b3698b4c17e17666d9e9a3c8eda2e78c4d9f8a4e443c10f985cb4cabf239b not found: ID does not exist" containerID="c53b3698b4c17e17666d9e9a3c8eda2e78c4d9f8a4e443c10f985cb4cabf239b"
Jul 29 12:11:55.362712 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:55.362632    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53b3698b4c17e17666d9e9a3c8eda2e78c4d9f8a4e443c10f985cb4cabf239b"} err="failed to get container status \"c53b3698b4c17e17666d9e9a3c8eda2e78c4d9f8a4e443c10f985cb4cabf239b\": rpc error: code = NotFound desc = could not find container \"c53b3698b4c17e17666d9e9a3c8eda2e78c4d9f8a4e443c10f985cb4cabf239b\": container with ID starting with c53b3698b4c17e17666d9e9a3c8eda2e78c4d9f8a4e443c10f985cb4cabf239b not found: ID does not exist"
Jul 29 12:11:55.374867 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:55.374836    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"]
Jul 29 12:11:55.380097 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:55.380073    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ddwlw"]
Jul 29 12:11:56.255343 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:56.255309    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05530424-c6ce-4729-a9fa-349d5173399c" path="/var/lib/kubelet/pods/05530424-c6ce-4729-a9fa-349d5173399c/volumes"
Jul 29 12:11:58.241796 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.241755    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hkldr/must-gather-5w2ht"]
Jul 29 12:11:58.242249 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.242231    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05530424-c6ce-4729-a9fa-349d5173399c" containerName="main"
Jul 29 12:11:58.242297 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.242251    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="05530424-c6ce-4729-a9fa-349d5173399c" containerName="main"
Jul 29 12:11:58.242297 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.242276    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f88da38-d6bd-41f4-9402-310428813647" containerName="main"
Jul 29 12:11:58.242297 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.242282    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f88da38-d6bd-41f4-9402-310428813647" containerName="main"
Jul 29 12:11:58.242390 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.242344    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f88da38-d6bd-41f4-9402-310428813647" containerName="main"
Jul 29 12:11:58.242390 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.242354    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="05530424-c6ce-4729-a9fa-349d5173399c" containerName="main"
Jul 29 12:11:58.247026 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.247000    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hkldr/must-gather-5w2ht"
Jul 29 12:11:58.249789 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.249764    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hkldr\"/\"kube-root-ca.crt\""
Jul 29 12:11:58.251069 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.251051    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hkldr\"/\"default-dockercfg-zq9sw\""
Jul 29 12:11:58.251182 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.251055    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hkldr\"/\"openshift-service-ca.crt\""
Jul 29 12:11:58.256226 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.256201    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hkldr/must-gather-5w2ht"]
Jul 29 12:11:58.342970 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.342932    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5srtj\" (UniqueName: \"kubernetes.io/projected/9b64e67d-e79d-4477-af5d-87eea0b608c0-kube-api-access-5srtj\") pod \"must-gather-5w2ht\" (UID: \"9b64e67d-e79d-4477-af5d-87eea0b608c0\") " pod="openshift-must-gather-hkldr/must-gather-5w2ht"
Jul 29 12:11:58.343303 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.343018    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b64e67d-e79d-4477-af5d-87eea0b608c0-must-gather-output\") pod \"must-gather-5w2ht\" (UID: \"9b64e67d-e79d-4477-af5d-87eea0b608c0\") " pod="openshift-must-gather-hkldr/must-gather-5w2ht"
Jul 29 12:11:58.444300 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.444265    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b64e67d-e79d-4477-af5d-87eea0b608c0-must-gather-output\") pod \"must-gather-5w2ht\" (UID: \"9b64e67d-e79d-4477-af5d-87eea0b608c0\") " pod="openshift-must-gather-hkldr/must-gather-5w2ht"
Jul 29 12:11:58.444484 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.444392    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5srtj\" (UniqueName: \"kubernetes.io/projected/9b64e67d-e79d-4477-af5d-87eea0b608c0-kube-api-access-5srtj\") pod \"must-gather-5w2ht\" (UID: \"9b64e67d-e79d-4477-af5d-87eea0b608c0\") " pod="openshift-must-gather-hkldr/must-gather-5w2ht"
Jul 29 12:11:58.444669 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.444648    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b64e67d-e79d-4477-af5d-87eea0b608c0-must-gather-output\") pod \"must-gather-5w2ht\" (UID: \"9b64e67d-e79d-4477-af5d-87eea0b608c0\") " pod="openshift-must-gather-hkldr/must-gather-5w2ht"
Jul 29 12:11:58.455504 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.455484    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5srtj\" (UniqueName: \"kubernetes.io/projected/9b64e67d-e79d-4477-af5d-87eea0b608c0-kube-api-access-5srtj\") pod \"must-gather-5w2ht\" (UID: \"9b64e67d-e79d-4477-af5d-87eea0b608c0\") " pod="openshift-must-gather-hkldr/must-gather-5w2ht"
Jul 29 12:11:58.557581 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.557551    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hkldr/must-gather-5w2ht"
Jul 29 12:11:58.689527 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.689479    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hkldr/must-gather-5w2ht"]
Jul 29 12:11:58.691810 ip-10-0-140-240 kubenswrapper[2584]: W0729 12:11:58.691773    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b64e67d_e79d_4477_af5d_87eea0b608c0.slice/crio-d257c1719ace8e54f1d5724fd37cab7e1e4d9762c5e1d3d91e3384643a118d5b WatchSource:0}: Error finding container d257c1719ace8e54f1d5724fd37cab7e1e4d9762c5e1d3d91e3384643a118d5b: Status 404 returned error can't find the container with id d257c1719ace8e54f1d5724fd37cab7e1e4d9762c5e1d3d91e3384643a118d5b
Jul 29 12:11:58.693616 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:58.693598    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 12:11:59.373940 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:11:59.373904    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hkldr/must-gather-5w2ht" event={"ID":"9b64e67d-e79d-4477-af5d-87eea0b608c0","Type":"ContainerStarted","Data":"d257c1719ace8e54f1d5724fd37cab7e1e4d9762c5e1d3d91e3384643a118d5b"}
Jul 29 12:12:05.418758 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:05.418639    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hkldr/must-gather-5w2ht" event={"ID":"9b64e67d-e79d-4477-af5d-87eea0b608c0","Type":"ContainerStarted","Data":"e4d24ce6ebfa8d3fcbea9d92b28b2a6c33367dc0966a08b867399bc87ee319ba"}
Jul 29 12:12:05.418758 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:05.418712    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hkldr/must-gather-5w2ht" event={"ID":"9b64e67d-e79d-4477-af5d-87eea0b608c0","Type":"ContainerStarted","Data":"605983aec372ed3cd7dea9440191bc503baaaea93c25dd19fcfceafed86d4138"}
Jul 29 12:12:05.437157 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:05.437103    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hkldr/must-gather-5w2ht" podStartSLOduration=1.029672118 podStartE2EDuration="7.437084418s" totalImagesPullingTime="6.4074123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 12:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 12:12:05.43640753 +0000 UTC m=+6193.629434878" watchObservedRunningTime="2026-07-29 12:12:05.437084418 +0000 UTC m=+6193.630111765"
Jul 29 12:12:21.900727 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:21.900616    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-xf9js_b053698e-b8cf-4b7e-80d1-ed373c08cfb4/istio-proxy/0.log"
Jul 29 12:12:21.957211 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:21.957158    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-xf9js_b053698e-b8cf-4b7e-80d1-ed373c08cfb4/istio-proxy/1.log"
Jul 29 12:12:23.099395 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:23.099361    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-xf9js_b053698e-b8cf-4b7e-80d1-ed373c08cfb4/istio-proxy/0.log"
Jul 29 12:12:23.156728 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:23.156687    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-xf9js_b053698e-b8cf-4b7e-80d1-ed373c08cfb4/istio-proxy/1.log"
Jul 29 12:12:24.288443 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:24.288401    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-xf9js_b053698e-b8cf-4b7e-80d1-ed373c08cfb4/istio-proxy/0.log"
Jul 29 12:12:24.343469 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:24.343429    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-xf9js_b053698e-b8cf-4b7e-80d1-ed373c08cfb4/istio-proxy/1.log"
Jul 29 12:12:25.531644 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:25.531608    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-xf9js_b053698e-b8cf-4b7e-80d1-ed373c08cfb4/istio-proxy/0.log"
Jul 29 12:12:25.587888 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:25.587853    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-xf9js_b053698e-b8cf-4b7e-80d1-ed373c08cfb4/istio-proxy/1.log"
Jul 29 12:12:26.835594 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:26.835559    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:28.356386 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:28.356355    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:29.596400 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:29.596370    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:30.826346 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:30.826317    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:32.041124 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:32.041081    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:33.260494 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:33.260467    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:34.503643 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:34.503610    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:35.710072 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:35.710031    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:36.932124 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:36.932087    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:38.162907 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:38.162875    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:39.385360 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:39.385331    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:40.640326 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:40.640297    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:41.836740 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:41.836711    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:43.044663 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:43.044625    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:44.275907 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:44.275876    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:45.513669 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:45.513636    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:46.739080 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:46.739029    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:47.990206 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:47.990176    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:49.221830 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:49.221795    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:50.452161 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:50.452127    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:51.689507 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:51.689481    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:52.925109 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:52.925072    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:54.140026 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:54.139991    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:55.346413 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:55.346372    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:56.568534 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:56.568488    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:12:57.696940 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:57.696907    2584 generic.go:356] "Generic (PLEG): container finished" podID="9b64e67d-e79d-4477-af5d-87eea0b608c0" containerID="605983aec372ed3cd7dea9440191bc503baaaea93c25dd19fcfceafed86d4138" exitCode=0
Jul 29 12:12:57.697445 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:57.696979    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hkldr/must-gather-5w2ht" event={"ID":"9b64e67d-e79d-4477-af5d-87eea0b608c0","Type":"ContainerDied","Data":"605983aec372ed3cd7dea9440191bc503baaaea93c25dd19fcfceafed86d4138"}
Jul 29 12:12:57.697445 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:57.697315    2584 scope.go:117] "RemoveContainer" containerID="605983aec372ed3cd7dea9440191bc503baaaea93c25dd19fcfceafed86d4138"
Jul 29 12:12:58.090236 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:12:58.090196    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hkldr_must-gather-5w2ht_9b64e67d-e79d-4477-af5d-87eea0b608c0/gather/0.log"
Jul 29 12:13:01.586744 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:01.586716    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nh6vt_cdea5088-94eb-4938-938f-ba83acfaa9d2/global-pull-secret-syncer/0.log"
Jul 29 12:13:01.818032 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:01.817992    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zkcxm_a0d8deb7-ef6f-4094-bdbe-b01ff9de1d6a/konnectivity-agent/0.log"
Jul 29 12:13:01.897114 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:01.897036    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-240.ec2.internal_6690eeea8150ab3c894966b36c641644/haproxy/0.log"
Jul 29 12:13:03.481200 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.481155    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-hkldr/must-gather-5w2ht"]
Jul 29 12:13:03.481725 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.481421    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-hkldr/must-gather-5w2ht" podUID="9b64e67d-e79d-4477-af5d-87eea0b608c0" containerName="copy" containerID="cri-o://e4d24ce6ebfa8d3fcbea9d92b28b2a6c33367dc0966a08b867399bc87ee319ba" gracePeriod=2
Jul 29 12:13:03.483975 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:13:03.483935    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5w2ht\" is forbidden: User \"system:node:ip-10-0-140-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hkldr\": no relationship found between node 'ip-10-0-140-240.ec2.internal' and this object" podUID="9b64e67d-e79d-4477-af5d-87eea0b608c0" pod="openshift-must-gather-hkldr/must-gather-5w2ht"
Jul 29 12:13:03.485265 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.485202    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-hkldr/must-gather-5w2ht"]
Jul 29 12:13:03.718636 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.718613    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hkldr_must-gather-5w2ht_9b64e67d-e79d-4477-af5d-87eea0b608c0/copy/0.log"
Jul 29 12:13:03.721213 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.721192    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hkldr/must-gather-5w2ht"
Jul 29 12:13:03.723520 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:13:03.723496    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5w2ht\" is forbidden: User \"system:node:ip-10-0-140-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hkldr\": no relationship found between node 'ip-10-0-140-240.ec2.internal' and this object" podUID="9b64e67d-e79d-4477-af5d-87eea0b608c0" pod="openshift-must-gather-hkldr/must-gather-5w2ht"
Jul 29 12:13:03.724843 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.724828    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hkldr_must-gather-5w2ht_9b64e67d-e79d-4477-af5d-87eea0b608c0/copy/0.log"
Jul 29 12:13:03.725167 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.725149    2584 generic.go:356] "Generic (PLEG): container finished" podID="9b64e67d-e79d-4477-af5d-87eea0b608c0" containerID="e4d24ce6ebfa8d3fcbea9d92b28b2a6c33367dc0966a08b867399bc87ee319ba" exitCode=143
Jul 29 12:13:03.725225 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.725185    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hkldr/must-gather-5w2ht"
Jul 29 12:13:03.725225 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.725192    2584 scope.go:117] "RemoveContainer" containerID="e4d24ce6ebfa8d3fcbea9d92b28b2a6c33367dc0966a08b867399bc87ee319ba"
Jul 29 12:13:03.727493 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:13:03.727473    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5w2ht\" is forbidden: User \"system:node:ip-10-0-140-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hkldr\": no relationship found between node 'ip-10-0-140-240.ec2.internal' and this object" podUID="9b64e67d-e79d-4477-af5d-87eea0b608c0" pod="openshift-must-gather-hkldr/must-gather-5w2ht"
Jul 29 12:13:03.732693 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.732653    2584 scope.go:117] "RemoveContainer" containerID="605983aec372ed3cd7dea9440191bc503baaaea93c25dd19fcfceafed86d4138"
Jul 29 12:13:03.746059 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.746036    2584 scope.go:117] "RemoveContainer" containerID="e4d24ce6ebfa8d3fcbea9d92b28b2a6c33367dc0966a08b867399bc87ee319ba"
Jul 29 12:13:03.746320 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:13:03.746301    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d24ce6ebfa8d3fcbea9d92b28b2a6c33367dc0966a08b867399bc87ee319ba\": container with ID starting with e4d24ce6ebfa8d3fcbea9d92b28b2a6c33367dc0966a08b867399bc87ee319ba not found: ID does not exist" containerID="e4d24ce6ebfa8d3fcbea9d92b28b2a6c33367dc0966a08b867399bc87ee319ba"
Jul 29 12:13:03.746429 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.746385    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d24ce6ebfa8d3fcbea9d92b28b2a6c33367dc0966a08b867399bc87ee319ba"} err="failed to get container status \"e4d24ce6ebfa8d3fcbea9d92b28b2a6c33367dc0966a08b867399bc87ee319ba\": rpc error: code = NotFound desc = could not find container \"e4d24ce6ebfa8d3fcbea9d92b28b2a6c33367dc0966a08b867399bc87ee319ba\": container with ID starting with e4d24ce6ebfa8d3fcbea9d92b28b2a6c33367dc0966a08b867399bc87ee319ba not found: ID does not exist"
Jul 29 12:13:03.746504 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.746447    2584 scope.go:117] "RemoveContainer" containerID="605983aec372ed3cd7dea9440191bc503baaaea93c25dd19fcfceafed86d4138"
Jul 29 12:13:03.746763 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:13:03.746737    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"605983aec372ed3cd7dea9440191bc503baaaea93c25dd19fcfceafed86d4138\": container with ID starting with 605983aec372ed3cd7dea9440191bc503baaaea93c25dd19fcfceafed86d4138 not found: ID does not exist" containerID="605983aec372ed3cd7dea9440191bc503baaaea93c25dd19fcfceafed86d4138"
Jul 29 12:13:03.746842 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.746766    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"605983aec372ed3cd7dea9440191bc503baaaea93c25dd19fcfceafed86d4138"} err="failed to get container status \"605983aec372ed3cd7dea9440191bc503baaaea93c25dd19fcfceafed86d4138\": rpc error: code = NotFound desc = could not find container \"605983aec372ed3cd7dea9440191bc503baaaea93c25dd19fcfceafed86d4138\": container with ID starting with 605983aec372ed3cd7dea9440191bc503baaaea93c25dd19fcfceafed86d4138 not found: ID does not exist"
Jul 29 12:13:03.858056 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.858025    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b64e67d-e79d-4477-af5d-87eea0b608c0-must-gather-output\") pod \"9b64e67d-e79d-4477-af5d-87eea0b608c0\" (UID: \"9b64e67d-e79d-4477-af5d-87eea0b608c0\") "
Jul 29 12:13:03.858220 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.858112    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5srtj\" (UniqueName: \"kubernetes.io/projected/9b64e67d-e79d-4477-af5d-87eea0b608c0-kube-api-access-5srtj\") pod \"9b64e67d-e79d-4477-af5d-87eea0b608c0\" (UID: \"9b64e67d-e79d-4477-af5d-87eea0b608c0\") "
Jul 29 12:13:03.860266 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.860236    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b64e67d-e79d-4477-af5d-87eea0b608c0-kube-api-access-5srtj" (OuterVolumeSpecName: "kube-api-access-5srtj") pod "9b64e67d-e79d-4477-af5d-87eea0b608c0" (UID: "9b64e67d-e79d-4477-af5d-87eea0b608c0"). InnerVolumeSpecName "kube-api-access-5srtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 12:13:03.864445 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.864415    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b64e67d-e79d-4477-af5d-87eea0b608c0-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "9b64e67d-e79d-4477-af5d-87eea0b608c0" (UID: "9b64e67d-e79d-4477-af5d-87eea0b608c0"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:13:03.959396 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.959365    2584 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b64e67d-e79d-4477-af5d-87eea0b608c0-must-gather-output\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 12:13:03.959396 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:03.959391    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5srtj\" (UniqueName: \"kubernetes.io/projected/9b64e67d-e79d-4477-af5d-87eea0b608c0-kube-api-access-5srtj\") on node \"ip-10-0-140-240.ec2.internal\" DevicePath \"\""
Jul 29 12:13:04.037154 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:13:04.037128    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5w2ht\" is forbidden: User \"system:node:ip-10-0-140-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hkldr\": no relationship found between node 'ip-10-0-140-240.ec2.internal' and this object" podUID="9b64e67d-e79d-4477-af5d-87eea0b608c0" pod="openshift-must-gather-hkldr/must-gather-5w2ht"
Jul 29 12:13:04.255702 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:04.255654    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b64e67d-e79d-4477-af5d-87eea0b608c0" path="/var/lib/kubelet/pods/9b64e67d-e79d-4477-af5d-87eea0b608c0/volumes"
Jul 29 12:13:06.180503 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:06.180471    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-fdkhh_9be44275-8cca-4b3e-ba47-3158d7e6be46/manager/0.log"
Jul 29 12:13:07.819804 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:07.819768    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_040367c1-1b23-442f-b75d-9df47bb63a34/alertmanager/0.log"
Jul 29 12:13:07.849808 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:07.849777    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_040367c1-1b23-442f-b75d-9df47bb63a34/config-reloader/0.log"
Jul 29 12:13:07.873007 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:07.872983    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_040367c1-1b23-442f-b75d-9df47bb63a34/kube-rbac-proxy-web/0.log"
Jul 29 12:13:07.899730 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:07.899708    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_040367c1-1b23-442f-b75d-9df47bb63a34/kube-rbac-proxy/0.log"
Jul 29 12:13:07.921315 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:07.921285    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_040367c1-1b23-442f-b75d-9df47bb63a34/kube-rbac-proxy-metric/0.log"
Jul 29 12:13:07.942810 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:07.942786    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_040367c1-1b23-442f-b75d-9df47bb63a34/prom-label-proxy/0.log"
Jul 29 12:13:07.964400 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:07.964373    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_040367c1-1b23-442f-b75d-9df47bb63a34/init-config-reloader/0.log"
Jul 29 12:13:08.171779 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:08.171752    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-d88rp_4d62d964-dccd-4dcb-9c9f-e07111b74671/monitoring-plugin/0.log"
Jul 29 12:13:08.205488 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:08.205463    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7tx4_ebff90ce-6282-4f61-a34d-9164b7c382ea/node-exporter/0.log"
Jul 29 12:13:08.227316 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:08.227293    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7tx4_ebff90ce-6282-4f61-a34d-9164b7c382ea/kube-rbac-proxy/0.log"
Jul 29 12:13:08.248400 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:08.248376    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7tx4_ebff90ce-6282-4f61-a34d-9164b7c382ea/init-textfile/0.log"
Jul 29 12:13:08.434497 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:08.434416    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rsxwn_3d27cf8b-53cf-45c5-8aba-302fd81d2364/kube-rbac-proxy-main/0.log"
Jul 29 12:13:08.456511 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:08.456482    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rsxwn_3d27cf8b-53cf-45c5-8aba-302fd81d2364/kube-rbac-proxy-self/0.log"
Jul 29 12:13:08.477008 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:08.476978    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rsxwn_3d27cf8b-53cf-45c5-8aba-302fd81d2364/openshift-state-metrics/0.log"
Jul 29 12:13:08.529655 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:08.529626    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c08b1a86-03e5-4374-9ab9-33d295ad57fc/prometheus/0.log"
Jul 29 12:13:08.551775 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:08.551732    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c08b1a86-03e5-4374-9ab9-33d295ad57fc/config-reloader/0.log"
Jul 29 12:13:08.567997 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:08.567969    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c08b1a86-03e5-4374-9ab9-33d295ad57fc/thanos-sidecar/0.log"
Jul 29 12:13:08.588771 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:08.588743    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c08b1a86-03e5-4374-9ab9-33d295ad57fc/kube-rbac-proxy-web/0.log"
Jul 29 12:13:08.609613 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:08.609588    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c08b1a86-03e5-4374-9ab9-33d295ad57fc/kube-rbac-proxy/0.log"
Jul 29 12:13:08.634812 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:08.634788    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c08b1a86-03e5-4374-9ab9-33d295ad57fc/kube-rbac-proxy-thanos/0.log"
Jul 29 12:13:08.655757 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:08.655730    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c08b1a86-03e5-4374-9ab9-33d295ad57fc/init-config-reloader/0.log"
Jul 29 12:13:12.172565 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.172532    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-bj476_7e6e3db5-2509-478a-989c-f26b47bf1f76/volume-data-source-validator/0.log"
Jul 29 12:13:12.283150 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.283114    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"]
Jul 29 12:13:12.283514 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.283489    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b64e67d-e79d-4477-af5d-87eea0b608c0" containerName="gather"
Jul 29 12:13:12.283598 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.283516    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b64e67d-e79d-4477-af5d-87eea0b608c0" containerName="gather"
Jul 29 12:13:12.283598 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.283530    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b64e67d-e79d-4477-af5d-87eea0b608c0" containerName="copy"
Jul 29 12:13:12.283598 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.283537    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b64e67d-e79d-4477-af5d-87eea0b608c0" containerName="copy"
Jul 29 12:13:12.283794 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.283647    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b64e67d-e79d-4477-af5d-87eea0b608c0" containerName="copy"
Jul 29 12:13:12.283794 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.283666    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b64e67d-e79d-4477-af5d-87eea0b608c0" containerName="gather"
Jul 29 12:13:12.289789 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.289767    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.292937 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.292913    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jjmjm\"/\"openshift-service-ca.crt\""
Jul 29 12:13:12.293036 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.292918    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jjmjm\"/\"kube-root-ca.crt\""
Jul 29 12:13:12.295111 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.295086    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jjmjm\"/\"default-dockercfg-zcqgd\""
Jul 29 12:13:12.297497 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.297466    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"]
Jul 29 12:13:12.319599 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.319569    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68da57bb-6fb7-4e19-baeb-b89a496a6a0f-proc\") pod \"perf-node-gather-daemonset-vmtds\" (UID: \"68da57bb-6fb7-4e19-baeb-b89a496a6a0f\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.319599 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.319601    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c65px\" (UniqueName: \"kubernetes.io/projected/68da57bb-6fb7-4e19-baeb-b89a496a6a0f-kube-api-access-c65px\") pod \"perf-node-gather-daemonset-vmtds\" (UID: \"68da57bb-6fb7-4e19-baeb-b89a496a6a0f\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.319799 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.319631    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68da57bb-6fb7-4e19-baeb-b89a496a6a0f-lib-modules\") pod \"perf-node-gather-daemonset-vmtds\" (UID: \"68da57bb-6fb7-4e19-baeb-b89a496a6a0f\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.319799 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.319725    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68da57bb-6fb7-4e19-baeb-b89a496a6a0f-sys\") pod \"perf-node-gather-daemonset-vmtds\" (UID: \"68da57bb-6fb7-4e19-baeb-b89a496a6a0f\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.319870 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.319816    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68da57bb-6fb7-4e19-baeb-b89a496a6a0f-podres\") pod \"perf-node-gather-daemonset-vmtds\" (UID: \"68da57bb-6fb7-4e19-baeb-b89a496a6a0f\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.420961 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.420919    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68da57bb-6fb7-4e19-baeb-b89a496a6a0f-podres\") pod \"perf-node-gather-daemonset-vmtds\" (UID: \"68da57bb-6fb7-4e19-baeb-b89a496a6a0f\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.420961 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.420961    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68da57bb-6fb7-4e19-baeb-b89a496a6a0f-proc\") pod \"perf-node-gather-daemonset-vmtds\" (UID: \"68da57bb-6fb7-4e19-baeb-b89a496a6a0f\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.421170 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.420978    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c65px\" (UniqueName: \"kubernetes.io/projected/68da57bb-6fb7-4e19-baeb-b89a496a6a0f-kube-api-access-c65px\") pod \"perf-node-gather-daemonset-vmtds\" (UID: \"68da57bb-6fb7-4e19-baeb-b89a496a6a0f\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.421170 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.421022    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68da57bb-6fb7-4e19-baeb-b89a496a6a0f-lib-modules\") pod \"perf-node-gather-daemonset-vmtds\" (UID: \"68da57bb-6fb7-4e19-baeb-b89a496a6a0f\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.421170 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.421074    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68da57bb-6fb7-4e19-baeb-b89a496a6a0f-proc\") pod \"perf-node-gather-daemonset-vmtds\" (UID: \"68da57bb-6fb7-4e19-baeb-b89a496a6a0f\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.421170 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.421094    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68da57bb-6fb7-4e19-baeb-b89a496a6a0f-sys\") pod \"perf-node-gather-daemonset-vmtds\" (UID: \"68da57bb-6fb7-4e19-baeb-b89a496a6a0f\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.421317 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.421169    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68da57bb-6fb7-4e19-baeb-b89a496a6a0f-sys\") pod \"perf-node-gather-daemonset-vmtds\" (UID: \"68da57bb-6fb7-4e19-baeb-b89a496a6a0f\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.421317 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.421201    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68da57bb-6fb7-4e19-baeb-b89a496a6a0f-podres\") pod \"perf-node-gather-daemonset-vmtds\" (UID: \"68da57bb-6fb7-4e19-baeb-b89a496a6a0f\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.421317 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.421213    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68da57bb-6fb7-4e19-baeb-b89a496a6a0f-lib-modules\") pod \"perf-node-gather-daemonset-vmtds\" (UID: \"68da57bb-6fb7-4e19-baeb-b89a496a6a0f\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.428498 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.428430    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c65px\" (UniqueName: \"kubernetes.io/projected/68da57bb-6fb7-4e19-baeb-b89a496a6a0f-kube-api-access-c65px\") pod \"perf-node-gather-daemonset-vmtds\" (UID: \"68da57bb-6fb7-4e19-baeb-b89a496a6a0f\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.601648 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.601614    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:12.732078 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.732054    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"]
Jul 29 12:13:12.735013 ip-10-0-140-240 kubenswrapper[2584]: W0729 12:13:12.734968    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod68da57bb_6fb7_4e19_baeb_b89a496a6a0f.slice/crio-1466fe667dac96b8fb8d5f694b1d6d29053120253ed63b751bf5a99f7ee1348c WatchSource:0}: Error finding container 1466fe667dac96b8fb8d5f694b1d6d29053120253ed63b751bf5a99f7ee1348c: Status 404 returned error can't find the container with id 1466fe667dac96b8fb8d5f694b1d6d29053120253ed63b751bf5a99f7ee1348c
Jul 29 12:13:12.765720 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:12.765646    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds" event={"ID":"68da57bb-6fb7-4e19-baeb-b89a496a6a0f","Type":"ContainerStarted","Data":"1466fe667dac96b8fb8d5f694b1d6d29053120253ed63b751bf5a99f7ee1348c"}
Jul 29 12:13:13.257124 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:13.257092    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vgmp2_74e2ee97-b7ae-45a8-a487-3061cc9bba14/dns/0.log"
Jul 29 12:13:13.276962 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:13.276932    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vgmp2_74e2ee97-b7ae-45a8-a487-3061cc9bba14/kube-rbac-proxy/0.log"
Jul 29 12:13:13.370035 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:13.370008    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9qhbp_605d5f71-2bcc-484f-a8fa-08cb75b43fe2/dns-node-resolver/0.log"
Jul 29 12:13:13.771714 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:13.771655    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds" event={"ID":"68da57bb-6fb7-4e19-baeb-b89a496a6a0f","Type":"ContainerStarted","Data":"e657db02b4e9d341c83db07911e530278b67bdcf10728db35be8044cd25ddebb"}
Jul 29 12:13:13.771922 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:13.771791    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:13.787398 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:13.787350    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds" podStartSLOduration=1.787334747 podStartE2EDuration="1.787334747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 12:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 12:13:13.786154208 +0000 UTC m=+6261.979181606" watchObservedRunningTime="2026-07-29 12:13:13.787334747 +0000 UTC m=+6261.980362094"
Jul 29 12:13:14.033851 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:14.033769    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b46vj_19da85ea-28f6-42ad-afc9-d37d9603cd90/node-ca/0.log"
Jul 29 12:13:15.391098 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:15.391065    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-xf9js_b053698e-b8cf-4b7e-80d1-ed373c08cfb4/istio-proxy/0.log"
Jul 29 12:13:15.442427 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:15.442393    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-xf9js_b053698e-b8cf-4b7e-80d1-ed373c08cfb4/istio-proxy/1.log"
Jul 29 12:13:16.015779 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:16.015746    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ggdhq_62d848b6-fe3c-489f-8f2d-af1a97db4032/serve-healthcheck-canary/0.log"
Jul 29 12:13:16.543217 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:16.543189    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hbvl8_617ed549-8c3d-45f4-8108-19aaf5d0fa2d/kube-rbac-proxy/0.log"
Jul 29 12:13:16.562304 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:16.562278    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hbvl8_617ed549-8c3d-45f4-8108-19aaf5d0fa2d/exporter/0.log"
Jul 29 12:13:16.582097 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:16.582057    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hbvl8_617ed549-8c3d-45f4-8108-19aaf5d0fa2d/extractor/0.log"
Jul 29 12:13:19.543994 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:19.543963    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5c847d9c7c-s7n7k_ba8b9c99-52d9-4b2a-8900-4a61dd0fe286/manager/0.log"
Jul 29 12:13:19.787472 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:19.787444    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-vmtds"
Jul 29 12:13:21.740120 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:21.740026    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-xf9js_b053698e-b8cf-4b7e-80d1-ed373c08cfb4/istio-proxy/0.log"
Jul 29 12:13:21.796504 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:21.796468    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-xf9js_b053698e-b8cf-4b7e-80d1-ed373c08cfb4/istio-proxy/1.log"
Jul 29 12:13:23.386969 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:23.386939    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-xf9js_b053698e-b8cf-4b7e-80d1-ed373c08cfb4/istio-proxy/0.log"
Jul 29 12:13:23.440373 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:23.440344    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-xf9js_b053698e-b8cf-4b7e-80d1-ed373c08cfb4/istio-proxy/1.log"
Jul 29 12:13:25.201839 ip-10-0-140-240 kubenswrapper[2584]: E0729 12:13:25.201794    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0c0efa00-4094-45d8-93e5-052c4139f31a"
Jul 29 12:13:25.825966 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:25.825934    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 29 12:13:26.645257 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:26.645225    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-b779r_f51faa87-2bae-442b-8d9e-01b4d270f0ec/migrator/0.log"
Jul 29 12:13:26.665323 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:26.665284    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-b779r_f51faa87-2bae-442b-8d9e-01b4d270f0ec/graceful-termination/0.log"
Jul 29 12:13:28.085265 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:28.085228    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8lmdp_97c7050d-3259-4926-83de-dfa81426daeb/kube-multus/0.log"
Jul 29 12:13:28.485231 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:28.485151    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpbv6_798afeb0-e754-4751-98dd-5dca2a693fe6/kube-multus-additional-cni-plugins/0.log"
Jul 29 12:13:28.504513 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:28.504489    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpbv6_798afeb0-e754-4751-98dd-5dca2a693fe6/egress-router-binary-copy/0.log"
Jul 29 12:13:28.523849 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:28.523821    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpbv6_798afeb0-e754-4751-98dd-5dca2a693fe6/cni-plugins/0.log"
Jul 29 12:13:28.568117 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:28.568081    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpbv6_798afeb0-e754-4751-98dd-5dca2a693fe6/bond-cni-plugin/0.log"
Jul 29 12:13:28.591826 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:28.591806    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpbv6_798afeb0-e754-4751-98dd-5dca2a693fe6/routeoverride-cni/0.log"
Jul 29 12:13:28.612505 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:28.612483    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpbv6_798afeb0-e754-4751-98dd-5dca2a693fe6/whereabouts-cni-bincopy/0.log"
Jul 29 12:13:28.634961 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:28.634924    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpbv6_798afeb0-e754-4751-98dd-5dca2a693fe6/whereabouts-cni/0.log"
Jul 29 12:13:28.659887 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:28.659859    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4wb9p_06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29/network-metrics-daemon/0.log"
Jul 29 12:13:28.676632 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:28.676608    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4wb9p_06a1f5e6-dc5b-41b1-af1f-c2fa5d510f29/kube-rbac-proxy/0.log"
Jul 29 12:13:29.640938 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:29.640907    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwrjz_154d9288-1e60-4c44-95cc-b4d0111a48ee/ovn-controller/0.log"
Jul 29 12:13:29.656081 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:29.656051    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwrjz_154d9288-1e60-4c44-95cc-b4d0111a48ee/ovn-acl-logging/0.log"
Jul 29 12:13:29.679736 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:29.679709    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwrjz_154d9288-1e60-4c44-95cc-b4d0111a48ee/kube-rbac-proxy-node/0.log"
Jul 29 12:13:29.700981 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:29.700954    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwrjz_154d9288-1e60-4c44-95cc-b4d0111a48ee/kube-rbac-proxy-ovn-metrics/0.log"
Jul 29 12:13:29.720059 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:29.720026    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwrjz_154d9288-1e60-4c44-95cc-b4d0111a48ee/northd/0.log"
Jul 29 12:13:29.739480 ip-10-0-140-240 kubenswrapper[2584]: I0729 12:13:29.739453    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bwrjz_154d9288-1e60-4c44-95cc-b4d0111a48ee/nbdb/0.log"