Jul 28 17:14:46.922743 ip-10-0-135-188 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 17:14:46.922835 ip-10-0-135-188 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 17:14:46.922845 ip-10-0-135-188 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 17:14:46.923169 ip-10-0-135-188 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 17:14:58.302805 ip-10-0-135-188 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 17:14:58.302822 ip-10-0-135-188 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 04605d53feca4b34ae440fd00c248574 --
Jul 28 17:16:43.618801 ip-10-0-135-188 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:16:44.055276 ip-10-0-135-188 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:16:44.055276 ip-10-0-135-188 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:16:44.055276 ip-10-0-135-188 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:16:44.055276 ip-10-0-135-188 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:16:44.055276 ip-10-0-135-188 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:16:44.056432 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.056344    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:16:44.061295 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061275    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:44.061295 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061292    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:44.061295 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061297    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:44.061295 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061301    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061306    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061312    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061316    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061322    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061326    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061333    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061337    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061341    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061345    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061349    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061353    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061356    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061362    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061368    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061374    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061378    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061382    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061387    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:44.061550 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061391    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061395    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061399    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061403    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061409    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061413    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061417    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061455    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061460    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061464    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061469    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061473    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061477    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061481    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061486    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061491    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061495    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061501    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061506    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061510    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:44.062331 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061514    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061537    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061554    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061558    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061562    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061568    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061575    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061580    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061586    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061590    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061594    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061599    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061603    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061607    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061611    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061616    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061620    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061624    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061628    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:44.063203 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061632    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061637    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061641    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061645    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061649    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061653    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061657    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061662    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061668    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061673    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061678    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061683    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061687    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061691    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061695    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061699    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061703    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061708    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061712    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061717    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:44.064051 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061721    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061725    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061729    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061733    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061737    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061741    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061745    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061749    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061753    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061757    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061762    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061766    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061772    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061777    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061782    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061787    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061791    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.061795    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.062940    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:44.064912 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.062950    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.062955    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.062960    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.062965    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.062970    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.062975    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.062979    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.062983    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.062987    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.062991    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.062996    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063000    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063004    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063009    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063013    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063017    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063021    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063026    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063030    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063034    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:44.065743 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063038    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063042    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063047    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063051    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063055    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063060    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063064    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063068    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063072    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063076    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063080    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063084    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063088    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063093    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063098    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063102    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063106    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063110    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063116    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063120    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:44.066472 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063124    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063129    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063133    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063137    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063141    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063145    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063149    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063153    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063157    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063161    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063168    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063174    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063179    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063184    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063188    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063193    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063198    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063202    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063208    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:44.067171 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063212    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063216    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063221    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063226    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063231    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063236    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063241    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063246    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063251    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063255    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063259    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063263    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063268    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063273    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063277    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063282    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063286    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063290    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063295    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063299    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:44.067786 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063303    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063309    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063315    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063319    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063323    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063328    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063334    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063338    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063342    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063346    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063350    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063354    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063358    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063363    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063367    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063373    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063379    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063384    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063390    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:44.068279 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063989    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.063998    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064003    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064008    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064013    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064019    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064023    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064028    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064032    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064037    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064041    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064045    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064049    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064054    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064057    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064061    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064065    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064069    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064073    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064077    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:44.068854 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064081    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064085    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064090    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064094    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064099    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064103    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064107    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064112    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064116    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064120    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064124    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064129    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064134    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064139    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064145    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064150    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064154    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064161    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064168    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:44.069348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064173    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064179    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064184    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064188    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064193    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064197    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064201    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064205    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064210    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064214    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064218    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064222    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064227    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064231    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064235    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064239    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064243    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064247    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064252    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064256    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:44.069845 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064261    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064265    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064270    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064274    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064278    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064282    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064286    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064290    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064295    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064299    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064303    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064310    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064320    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064326    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064330    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064335    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064339    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064344    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064348    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064352    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:44.070329 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064357    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064361    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064365    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064369    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064373    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064377    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064381    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064385    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064390    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064395    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064399    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064403    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064407    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064412    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064416    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064420    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064424    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064428    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064432    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.064437    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:44.070832 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064558    2575 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064570    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064580    2575 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064587    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064593    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064609    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064616    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064622    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064627    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064632    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064638    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064643    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064648    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064653    2575 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064658    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064663    2575 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064668    2575 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064673    2575 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064679    2575 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064684    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064689    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064694    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064699    2575 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064705    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:16:44.071315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064713    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064719    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064724    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064730    2575 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064735    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064739    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064744    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064749    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064756    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064761    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064766    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064770    2575 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064775    2575 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064780    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064788    2575 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064793    2575 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064798    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064803    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064808    2575 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064814    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064819    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064824    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064829    2575 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064834    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064839    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:16:44.071989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064843    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064849    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064853    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064858    2575 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064862    2575 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064868    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064873    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064880    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064885    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064891    2575 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064895    2575 flags.go:64] FLAG: --help="false"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064902    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064907    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064911    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064916    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064921    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064927    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064932    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064936    2575 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064941    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064946    2575 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064951    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064957    2575 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064962    2575 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:16:44.072691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064966    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064971    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064976    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064980    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064985    2575 flags.go:64] FLAG: --lock-file=""
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064989    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064994    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.064999    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065008    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065012    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065017    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065021    2575 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065026    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065031    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065036    2575 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065042    2575 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065049    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065054    2575 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065060    2575 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065067    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065072    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065077    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065081    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065086    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065091    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:16:44.073487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065096    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065106    2575 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065111    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065117    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065121    2575 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065126    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065135    2575 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065140    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065145    2575 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065150    2575 flags.go:64] FLAG: --port="10250"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065155    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065159    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07a4913ddabdb59fe"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065165    2575 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065169    2575 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065174    2575 flags.go:64] FLAG: --register-node="true"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065178    2575 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065184    2575 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065189    2575 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065193    2575 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065198    2575 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065204    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065209    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065214    2575 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065220    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065225    2575 flags.go:64] FLAG: --runonce="false"
Jul 28 17:16:44.074260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065229    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065236    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065241    2575 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065246    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065251    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065256    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065261    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065266    2575 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065271    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065275    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065280    2575 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065285    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065290    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065295    2575 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065300    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065308    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065313    2575 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065317    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065324    2575 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065328    2575 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065333    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065338    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065343    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065347    2575 flags.go:64] FLAG: --v="2"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065353    2575 flags.go:64] FLAG: --version="false"
Jul 28 17:16:44.074978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065359    2575 flags.go:64] FLAG: --vmodule=""
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065365    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.065371    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065568    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065575    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065582    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065587    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065591    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065600    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065605    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065609    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065613    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065617    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065621    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065626    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065631    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065635    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065640    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065644    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065648    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:44.075596 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065652    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065656    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065661    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065666    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065671    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065675    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065679    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065683    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065688    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065692    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065696    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065701    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065705    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065711    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065718    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065725    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065731    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065739    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065744    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:44.076094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065748    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065753    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065758    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065762    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065767    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065772    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065776    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065781    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065785    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065789    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065793    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065797    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065802    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065806    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065810    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065815    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065820    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065824    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065828    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065833    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:44.076604 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065837    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065841    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065845    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065851    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065857    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065862    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065867    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065872    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065876    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065880    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065886    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065891    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065895    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065900    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065904    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065909    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065914    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065918    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065922    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065926    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:44.077122 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065931    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065935    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065939    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065944    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065948    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065952    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065957    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065961    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065965    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065970    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065974    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065979    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065983    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065987    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065992    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.065996    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.066001    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.066005    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.066009    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.066013    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:44.077637 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.066017    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:44.078126 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.066021    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:44.078126 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.066028    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:44.078126 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.066037    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:16:44.078126 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.074417    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:16:44.078126 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.074432    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:16:44.078126 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.075327    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:16:44.078126 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.075335    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:16:44.078126 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075392    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:44.078126 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075397    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:44.078126 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075401    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:44.078126 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075405    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:44.078126 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075409    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:44.078126 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075413    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:44.078126 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075416    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075419    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075422    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075425    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075428    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075431    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075434    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075437    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075440    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075443    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075446    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075448    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075451    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075453    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075456    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075459    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075462    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075464    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075467    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:44.078475 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075469    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075474    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075477    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075480    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075484    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075487    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075490    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075492    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075495    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075498    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075500    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075503    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075506    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075509    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075511    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075514    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075517    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075532    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075535    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075538    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:44.078985 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075540    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075543    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075546    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075549    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075552    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075554    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075557    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075559    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075562    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075564    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075567    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075570    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075572    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075575    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075578    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075580    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075583    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075586    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075588    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075591    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:44.079467 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075594    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075596    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075599    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075601    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075604    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075607    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075610    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075612    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075615    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075618    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075622    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075625    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075628    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075631    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075635    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075637    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075640    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075642    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075645    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075648    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:44.079980 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075650    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:44.080517 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075652    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:44.080517 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075655    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:44.080517 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075657    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:44.080517 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075660    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:44.080517 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075662    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:44.080517 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075665    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:44.080517 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075667    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:44.080517 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075670    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:44.080517 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075673    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:44.080517 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075676    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:44.080517 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075678    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:44.080517 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075681    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:44.080517 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075683    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:44.080517 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.075689    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:16:44.080517 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075815    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075820    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075823    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075826    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075829    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075831    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075834    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075836    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075839    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075843    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075847    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075849    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075852    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075855    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075858    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075860    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075863    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075865    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075868    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075871    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:44.080976 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075873    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075876    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075879    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075881    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075884    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075887    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075889    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075891    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075894    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075896    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075899    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075902    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075904    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075907    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075909    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075912    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075914    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075917    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075919    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075922    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:44.081460 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075924    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075927    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075929    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075932    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075934    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075937    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075940    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075943    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075945    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075949    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075952    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075956    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075959    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075961    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075964    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075967    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075970    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075972    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075975    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:44.081978 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075977    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075980    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075982    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075986    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075989    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075992    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075995    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.075998    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076000    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076003    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076006    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076008    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076011    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076014    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076016    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076019    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076022    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076025    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076027    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:44.082466 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076030    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076033    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076036    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076038    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076041    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076043    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076045    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076048    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076051    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076053    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076057    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076059    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076061    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076064    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076066    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076069    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076071    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076074    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076077    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076079    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:44.082960 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:44.076081    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:44.083438 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.076086    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:16:44.083438 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.076186    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:16:44.083438 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.078630    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:16:44.083438 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.080064    2575 server.go:1019] "Starting client certificate rotation"
Jul 28 17:16:44.083438 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.080161    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:16:44.083438 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.080198    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:16:44.083438 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.082467    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:16:44.086076 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.086062    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:16:44.096710 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.096693    2575 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:16:44.099299 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.099280    2575 log.go:25] "Validated CRI v1 image API"
Jul 28 17:16:44.100146 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.100132    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:16:44.104652 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.104628    2575 fs.go:135] Filesystem UUIDs: map[36ff76ce-77d0-4580-822b-68d1e6cc9d53:/dev/nvme0n1p4 758664f8-d7dd-4115-9570-e2d34848caec:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 28 17:16:44.104710 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.104650    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:16:44.110393 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.110288    2575 manager.go:216] Machine: {Timestamp:2026-07-28 17:16:44.108898096 +0000 UTC m=+0.366507776 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098407 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec206d918a2aa14305c4c8700f5f252c SystemUUID:ec206d91-8a2a-a143-05c4-c8700f5f252c BootID:04605d53-feca-4b34-ae44-0fd00c248574 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:38:f6:81:dd:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:38:f6:81:dd:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:aa:e8:a7:34:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:16:44.110393 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.110383    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:16:44.110531 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.110486    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:16:44.111138 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.111122    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:16:44.113822 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.113793    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:16:44.113966 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.113826    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-188.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:16:44.114596 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.114587    2575 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:16:44.114627 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.114606    2575 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:16:44.114627 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.114614    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:16:44.115648 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.115637    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:16:44.116906 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.116895    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:16:44.119856 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.119845    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:16:44.119966 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.119957    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:16:44.119995 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.119986    2575 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:16:44.120025 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.119995    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:16:44.120025 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.120009    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:16:44.120025 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.120018    2575 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:16:44.120122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.120027    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:16:44.120485 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.120446    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:16:44.120485 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.120481    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:16:44.124300 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.124265    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:16:44.126390 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.126375    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:16:44.126434 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.126424    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:16:44.128055 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.128039    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:16:44.128101 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.128067    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:16:44.128101 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.128078    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:16:44.128101 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.128086    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:16:44.128101 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.128095    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:16:44.128101 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.128101    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:16:44.128234 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.128108    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:16:44.128234 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.128114    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:16:44.128234 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.128120    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:16:44.128234 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.128125    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:16:44.128234 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.128133    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:16:44.128234 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.128138    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:16:44.129834 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.129821    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:16:44.129867 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.129837    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:16:44.130036 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.130010    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-188.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:16:44.130105 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.130047    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:16:44.131582 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.131562    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-188.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:16:44.133112 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.133099    2575 server.go:1268] "Started kubelet"
Jul 28 17:16:44.133189 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.133162    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:16:44.133266 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.133168    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:16:44.133333 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.133315    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:16:44.133938 ip-10-0-135-188 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:16:44.134417 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.134399    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:16:44.137962 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.137948    2575 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:16:44.139680 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.139660    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:16:44.143138 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.142261    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-188.ec2.internal.18c6839b056b5663  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-188.ec2.internal,UID:ip-10-0-135-188.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-188.ec2.internal,},FirstTimestamp:2026-07-28 17:16:44.133070435 +0000 UTC m=+0.390680116,LastTimestamp:2026-07-28 17:16:44.133070435 +0000 UTC m=+0.390680116,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-188.ec2.internal,}"
Jul 28 17:16:44.143239 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.143156    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:16:44.143239 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.143165    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:16:44.143339 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.143251    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 28 17:16:44.143717 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.143702    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:16:44.143868 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.143847    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:16:44.143868 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.143868    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:16:44.144030 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.143962    2575 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:16:44.144030 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.143971    2575 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:16:44.144176 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.144157    2575 factory.go:55] Registering systemd factory
Jul 28 17:16:44.144279 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.144180    2575 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:16:44.144456 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.144440    2575 factory.go:153] Registering CRI-O factory
Jul 28 17:16:44.144456 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.144455    2575 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:16:44.144633 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.144499    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:16:44.144633 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.144577    2575 factory.go:103] Registering Raw factory
Jul 28 17:16:44.144633 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.144606    2575 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:16:44.146004 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.145436    2575 manager.go:349] Starting recovery of all containers
Jul 28 17:16:44.146827 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.146794    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-188.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:16:44.146941 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.146917    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:16:44.159386 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.159366    2575 manager.go:354] Recovery completed
Jul 28 17:16:44.163351 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.163339    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:44.164769 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.164750    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hncsk"
Jul 28 17:16:44.165132 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.165116    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:44.165201 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.165151    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:44.165201 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.165167    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:44.165734 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.165718    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:16:44.165734 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.165734    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:16:44.165844 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.165722    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:44.165844 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.165752    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:16:44.166485 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.166470    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:44.166577 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.166499    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:44.166577 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.166514    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:44.166797 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.166724    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-188.ec2.internal.18c6839b075495e6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-188.ec2.internal,UID:ip-10-0-135-188.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-188.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-188.ec2.internal,},FirstTimestamp:2026-07-28 17:16:44.165133798 +0000 UTC m=+0.422743482,LastTimestamp:2026-07-28 17:16:44.165133798 +0000 UTC m=+0.422743482,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-188.ec2.internal,}"
Jul 28 17:16:44.167483 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.167470    2575 policy_none.go:49] "None policy: Start"
Jul 28 17:16:44.167561 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.167485    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:16:44.167561 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.167496    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:16:44.168608 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.168587    2575 policy_none.go:47] "Start"
Jul 28 17:16:44.171643 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.171457    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hncsk"
Jul 28 17:16:44.177279 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.177202    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-188.ec2.internal.18c6839b0754f729  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-188.ec2.internal,UID:ip-10-0-135-188.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-188.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-188.ec2.internal,},FirstTimestamp:2026-07-28 17:16:44.165158697 +0000 UTC m=+0.422768382,LastTimestamp:2026-07-28 17:16:44.165158697 +0000 UTC m=+0.422768382,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-188.ec2.internal,}"
Jul 28 17:16:44.200128 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.200114    2575 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:16:44.200199 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.200143    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:16:44.200199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.200153    2575 server.go:87] "Starting device plugin registration server"
Jul 28 17:16:44.200435 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.200422    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:16:44.200471 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.200441    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:16:44.200563 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.200541    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:16:44.200703 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.200687    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:16:44.200703 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.200701    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:16:44.201227 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.201204    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:16:44.201303 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.201245    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 28 17:16:44.221903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.221886    2575 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2609 into the system container \"/system.slice\": <nil>"
Jul 28 17:16:44.307112 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.300970    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:44.307112 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.301903    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:44.307112 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.301935    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:44.307112 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.301948    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:44.307112 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.301979    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.307112 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.304516    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:16:44.307112 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.305618    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:16:44.307112 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.305652    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:16:44.307112 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.305671    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:16:44.307112 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.305704    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:16:44.308126 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.308100    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:44.311091 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.311076    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.311155 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.311099    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-188.ec2.internal\": node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 28 17:16:44.363109 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.363083    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 28 17:16:44.406190 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.406160    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal"]
Jul 28 17:16:44.406287 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.406251    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:44.407824 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.407807    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:44.407937 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.407837    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:44.407937 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.407847    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:44.409164 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.409152    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:44.409279 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.409265    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.409333 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.409301    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:44.409901 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.409883    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:44.409993 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.409912    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:44.409993 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.409927    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:44.409993 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.409885    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:44.409993 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.409990    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:44.410172 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.410001    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:44.411760 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.411744    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.411803 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.411772    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:44.412371 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.412355    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:44.412453 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.412383    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:44.412453 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.412395    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:44.439060 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.439045    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-188.ec2.internal\" not found" node="ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.442244 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.442228    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-188.ec2.internal\" not found" node="ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.445655 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.445639    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44dbed9217caa115cd6a389d801add6e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal\" (UID: \"44dbed9217caa115cd6a389d801add6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.445735 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.445668    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44dbed9217caa115cd6a389d801add6e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal\" (UID: \"44dbed9217caa115cd6a389d801add6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.445735 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.445694    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/11997b299c0cb2944eaf7c6feeeb8aab-config\") pod \"kube-apiserver-proxy-ip-10-0-135-188.ec2.internal\" (UID: \"11997b299c0cb2944eaf7c6feeeb8aab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.463816 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.463797    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 28 17:16:44.546483 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.546460    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44dbed9217caa115cd6a389d801add6e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal\" (UID: \"44dbed9217caa115cd6a389d801add6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.546604 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.546486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/11997b299c0cb2944eaf7c6feeeb8aab-config\") pod \"kube-apiserver-proxy-ip-10-0-135-188.ec2.internal\" (UID: \"11997b299c0cb2944eaf7c6feeeb8aab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.546604 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.546503    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44dbed9217caa115cd6a389d801add6e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal\" (UID: \"44dbed9217caa115cd6a389d801add6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.546604 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.546538    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44dbed9217caa115cd6a389d801add6e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal\" (UID: \"44dbed9217caa115cd6a389d801add6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.546604 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.546569    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44dbed9217caa115cd6a389d801add6e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal\" (UID: \"44dbed9217caa115cd6a389d801add6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.546604 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.546574    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/11997b299c0cb2944eaf7c6feeeb8aab-config\") pod \"kube-apiserver-proxy-ip-10-0-135-188.ec2.internal\" (UID: \"11997b299c0cb2944eaf7c6feeeb8aab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.564624 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.564570    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 28 17:16:44.665342 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.665312    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 28 17:16:44.741528 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.741502    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.746632 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.746612    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal"
Jul 28 17:16:44.766368 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.766343    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 28 17:16:44.867017 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:44.866984    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-188.ec2.internal\" not found"
Jul 28 17:16:44.947673 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:44.947648    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:45.043850 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.043809    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal"
Jul 28 17:16:45.047513 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.046171    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:45.057050 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.057026    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:16:45.058189 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.058174    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal"
Jul 28 17:16:45.064487 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.064468    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:16:45.079846 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.079787    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:16:45.079949 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.079926    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:16:45.079992 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.079926    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:16:45.079992 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.079935    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:16:45.120332 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.120299    2575 apiserver.go:52] "Watching apiserver"
Jul 28 17:16:45.134680 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.134657    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:16:45.135048 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.135022    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-vvdjg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6","openshift-dns/node-resolver-gcwqn","openshift-image-registry/node-ca-wptfc","openshift-multus/multus-additional-cni-plugins-xnpnk","openshift-multus/multus-l4s6b","openshift-network-diagnostics/network-check-target-n6cbg","openshift-ovn-kubernetes/ovnkube-node-c954x","kube-system/global-pull-secret-syncer-8sv5v","kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal","openshift-cluster-node-tuning-operator/tuned-8f9mp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal","openshift-multus/network-metrics-daemon-zqjzm","openshift-network-operator/iptables-alerter-7hs9z"]
Jul 28 17:16:45.136803 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.136781    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.138388 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.138369    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7hs9z"
Jul 28 17:16:45.139174 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.139155    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:16:45.139263 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.139175    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:16:45.139263 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.139218    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:16:45.139403 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.139383    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:16:45.139449 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.139393    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pch5h\""
Jul 28 17:16:45.139536 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.139484    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gcwqn"
Jul 28 17:16:45.140583 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.140566    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:16:45.140683 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.140581    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:16:45.140683 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.140570    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:16:45.140838 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.140825    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-58r5s\""
Jul 28 17:16:45.141035 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.141018    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vvdjg"
Jul 28 17:16:45.141412 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.141396    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:16:45.141473 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.141423    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:16:45.141650 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.141636    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ntdq4\""
Jul 28 17:16:45.142230 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.142217    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:16:45.142286 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.142267    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:16:45.143228 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.143212    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:16:45.143314 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.143238    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z9r66\""
Jul 28 17:16:45.143314 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.143258    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:16:45.143314 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.143239    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:16:45.143717 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.143694    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wptfc"
Jul 28 17:16:45.145113 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.145096    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.145564 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.145547    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:16:45.145813 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.145799    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rspxk\""
Jul 28 17:16:45.145813 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.145808    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:16:45.145924 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.145867    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:16:45.146493 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.146478    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.146996 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.146974    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:16:45.147096 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.147027    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:16:45.147173 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.147160    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g55cr\""
Jul 28 17:16:45.147611 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.147595    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:45.147673 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.147649    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:16:45.148637 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.148620    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r6lkq\""
Jul 28 17:16:45.148721 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.148654    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:16:45.148721 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.148710    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:16:45.149015 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.149000    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.149182 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.149164    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:16:45.149271 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.149242    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:16:45.149451 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.149435    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:16:45.149506 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.149453    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-etc-kubernetes\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.149506 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.149473    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:16:45.149594 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.149512    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4350004-c24b-49f4-9142-b7d268dd1132-host\") pod \"node-ca-wptfc\" (UID: \"f4350004-c24b-49f4-9142-b7d268dd1132\") " pod="openshift-image-registry/node-ca-wptfc"
Jul 28 17:16:45.149594 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.149542    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/325554ac-10f4-4076-b562-20b27ddf18b0-system-cni-dir\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.149697 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.149609    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/325554ac-10f4-4076-b562-20b27ddf18b0-cnibin\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.149697 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.149643    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/325554ac-10f4-4076-b562-20b27ddf18b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.149697 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.149674    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-cnibin\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.150054 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.150012    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-hostroot\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.150134 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.150075    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-multus-conf-dir\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.150197 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.150134    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae4f1d2d-d42f-445d-aa5a-88659cbf4598-iptables-alerter-script\") pod \"iptables-alerter-7hs9z\" (UID: \"ae4f1d2d-d42f-445d-aa5a-88659cbf4598\") " pod="openshift-network-operator/iptables-alerter-7hs9z"
Jul 28 17:16:45.150520 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.150476    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae4f1d2d-d42f-445d-aa5a-88659cbf4598-host-slash\") pod \"iptables-alerter-7hs9z\" (UID: \"ae4f1d2d-d42f-445d-aa5a-88659cbf4598\") " pod="openshift-network-operator/iptables-alerter-7hs9z"
Jul 28 17:16:45.150625 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.150601    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:45.150754 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.150680    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:16:45.150901 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.150829    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv2vs\" (UniqueName: \"kubernetes.io/projected/f4350004-c24b-49f4-9142-b7d268dd1132-kube-api-access-rv2vs\") pod \"node-ca-wptfc\" (UID: \"f4350004-c24b-49f4-9142-b7d268dd1132\") " pod="openshift-image-registry/node-ca-wptfc"
Jul 28 17:16:45.150901 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.150880    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bac64312-87a3-4c2c-8cfa-80754d0123ed-hosts-file\") pod \"node-resolver-gcwqn\" (UID: \"bac64312-87a3-4c2c-8cfa-80754d0123ed\") " pod="openshift-dns/node-resolver-gcwqn"
Jul 28 17:16:45.151036 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.150920    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5shf\" (UniqueName: \"kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf\") pod \"network-check-target-n6cbg\" (UID: \"e041de26-66e3-4855-9270-cf54fe1d8045\") " pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:16:45.151036 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.150954    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-multus-cni-dir\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.151036 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.150981    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42x4d\" (UniqueName: \"kubernetes.io/projected/beefa577-daaa-474f-9acf-80700f353213-kube-api-access-42x4d\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.151036 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.151016    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ab5d559-43a9-4c98-9576-48e592b09677-konnectivity-ca\") pod \"konnectivity-agent-vvdjg\" (UID: \"1ab5d559-43a9-4c98-9576-48e592b09677\") " pod="kube-system/konnectivity-agent-vvdjg"
Jul 28 17:16:45.151233 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.151058    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsnwf\" (UniqueName: \"kubernetes.io/projected/bac64312-87a3-4c2c-8cfa-80754d0123ed-kube-api-access-vsnwf\") pod \"node-resolver-gcwqn\" (UID: \"bac64312-87a3-4c2c-8cfa-80754d0123ed\") " pod="openshift-dns/node-resolver-gcwqn"
Jul 28 17:16:45.151233 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.151184    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-system-cni-dir\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.151233 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.151223    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-multus-socket-dir-parent\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.151399 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.151252    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/beefa577-daaa-474f-9acf-80700f353213-multus-daemon-config\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.151399 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.151288    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkbw5\" (UniqueName: \"kubernetes.io/projected/ae4f1d2d-d42f-445d-aa5a-88659cbf4598-kube-api-access-jkbw5\") pod \"iptables-alerter-7hs9z\" (UID: \"ae4f1d2d-d42f-445d-aa5a-88659cbf4598\") " pod="openshift-network-operator/iptables-alerter-7hs9z"
Jul 28 17:16:45.151399 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.151331    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ab5d559-43a9-4c98-9576-48e592b09677-agent-certs\") pod \"konnectivity-agent-vvdjg\" (UID: \"1ab5d559-43a9-4c98-9576-48e592b09677\") " pod="kube-system/konnectivity-agent-vvdjg"
Jul 28 17:16:45.151399 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.151384    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac64312-87a3-4c2c-8cfa-80754d0123ed-tmp-dir\") pod \"node-resolver-gcwqn\" (UID: \"bac64312-87a3-4c2c-8cfa-80754d0123ed\") " pod="openshift-dns/node-resolver-gcwqn"
Jul 28 17:16:45.151617 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.151426    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/325554ac-10f4-4076-b562-20b27ddf18b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.151617 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.151455    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-run-k8s-cni-cncf-io\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.151966 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.151386    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mtnmv\""
Jul 28 17:16:45.152162 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.152142    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-run-netns\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.152237 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.152192    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-var-lib-cni-multus\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.152237 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.152228    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-run-multus-certs\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.152356 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.152238    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:16:45.152356 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.152267    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/325554ac-10f4-4076-b562-20b27ddf18b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.152356 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.152302    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-var-lib-kubelet\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.152356 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.152332    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4350004-c24b-49f4-9142-b7d268dd1132-serviceca\") pod \"node-ca-wptfc\" (UID: \"f4350004-c24b-49f4-9142-b7d268dd1132\") " pod="openshift-image-registry/node-ca-wptfc"
Jul 28 17:16:45.152567 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.152366    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmg6v\" (UniqueName: \"kubernetes.io/projected/325554ac-10f4-4076-b562-20b27ddf18b0-kube-api-access-xmg6v\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.152567 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.152403    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-os-release\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.152700 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.152676    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-var-lib-cni-bin\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.152813 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.152713    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/325554ac-10f4-4076-b562-20b27ddf18b0-os-release\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.152813 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.152729    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:16:45.152813 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.152733    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/325554ac-10f4-4076-b562-20b27ddf18b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.152813 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.152793    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/beefa577-daaa-474f-9acf-80700f353213-cni-binary-copy\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.153838 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.153467    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.154601 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.154577    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:16:45.155702 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.155688    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:16:45.155797 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.155777    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:16:45.155889 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.155875    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t8p6k\""
Jul 28 17:16:45.156505 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.156363    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:16:45.181251 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.176554    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:11:44 +0000 UTC" deadline="2028-02-17 13:49:18.095233808 +0000 UTC"
Jul 28 17:16:45.181251 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.176604    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13652h32m32.918633855s"
Jul 28 17:16:45.186394 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.186369    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ljr5n"
Jul 28 17:16:45.196974 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.196949    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ljr5n"
Jul 28 17:16:45.244941 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.244914    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:16:45.253770 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.253743    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-run-ovn\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.253886 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.253790    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/325554ac-10f4-4076-b562-20b27ddf18b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.253886 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.253815    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-cnibin\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.253886 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.253833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-hostroot\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.253886 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.253848    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-multus-conf-dir\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.253886 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.253869    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-env-overrides\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.254100 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.253893    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-cni-bin\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.254100 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.253918    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-hostroot\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.254100 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.253939    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-multus-conf-dir\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.254100 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.253960    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-cnibin\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.254100 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.253965    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv2vs\" (UniqueName: \"kubernetes.io/projected/f4350004-c24b-49f4-9142-b7d268dd1132-kube-api-access-rv2vs\") pod \"node-ca-wptfc\" (UID: \"f4350004-c24b-49f4-9142-b7d268dd1132\") " pod="openshift-image-registry/node-ca-wptfc"
Jul 28 17:16:45.254100 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254002    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-kubernetes\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.254100 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254048    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ab5d559-43a9-4c98-9576-48e592b09677-konnectivity-ca\") pod \"konnectivity-agent-vvdjg\" (UID: \"1ab5d559-43a9-4c98-9576-48e592b09677\") " pod="kube-system/konnectivity-agent-vvdjg"
Jul 28 17:16:45.254100 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254068    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-cni-netd\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.254100 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254088    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ab5d559-43a9-4c98-9576-48e592b09677-agent-certs\") pod \"konnectivity-agent-vvdjg\" (UID: \"1ab5d559-43a9-4c98-9576-48e592b09677\") " pod="kube-system/konnectivity-agent-vvdjg"
Jul 28 17:16:45.254575 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac64312-87a3-4c2c-8cfa-80754d0123ed-tmp-dir\") pod \"node-resolver-gcwqn\" (UID: \"bac64312-87a3-4c2c-8cfa-80754d0123ed\") " pod="openshift-dns/node-resolver-gcwqn"
Jul 28 17:16:45.254575 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254133    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-systemd-units\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.254575 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254178    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-run-systemd\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.254575 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254208    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-ovnkube-config\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.254575 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254250    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9qwq\" (UniqueName: \"kubernetes.io/projected/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-kube-api-access-l9qwq\") pod \"network-metrics-daemon-zqjzm\" (UID: \"0ee703af-bd7c-45e1-8b8a-1b31645d40bf\") " pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:45.254575 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254288    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/325554ac-10f4-4076-b562-20b27ddf18b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.254575 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254315    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-run-multus-certs\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.254575 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254370    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-slash\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.254575 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254376    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-run-multus-certs\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.254575 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254407    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/325554ac-10f4-4076-b562-20b27ddf18b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.254575 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254471    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-log-socket\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.254575 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254496    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2de9f57b-63bf-4dad-be1a-86716a7758f8-dbus\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:45.254575 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254492    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac64312-87a3-4c2c-8cfa-80754d0123ed-tmp-dir\") pod \"node-resolver-gcwqn\" (UID: \"bac64312-87a3-4c2c-8cfa-80754d0123ed\") " pod="openshift-dns/node-resolver-gcwqn"
Jul 28 17:16:45.254575 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254545    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-systemd\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254579    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-var-lib-kubelet\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254611    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254658    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-etc-openvswitch\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254662    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-var-lib-kubelet\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254684    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-run\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254722    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ef16b68-fef4-40d7-8ca2-66b900480734-tmp\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254741    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ab5d559-43a9-4c98-9576-48e592b09677-konnectivity-ca\") pod \"konnectivity-agent-vvdjg\" (UID: \"1ab5d559-43a9-4c98-9576-48e592b09677\") " pod="kube-system/konnectivity-agent-vvdjg"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254747    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-os-release\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254774    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-var-lib-cni-bin\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254811    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-kubelet\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254836    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-sysconfig\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254843    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-os-release\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254843    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-var-lib-cni-bin\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254880    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-registration-dir\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254905    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-sys-fs\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254907    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/325554ac-10f4-4076-b562-20b27ddf18b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.255425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254924    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/325554ac-10f4-4076-b562-20b27ddf18b0-os-release\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254948    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/325554ac-10f4-4076-b562-20b27ddf18b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254984    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-etc-kubernetes\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.254990    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/325554ac-10f4-4076-b562-20b27ddf18b0-os-release\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255000    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-ovn-node-metrics-cert\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255034    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-etc-kubernetes\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255023    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-var-lib-kubelet\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255078    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz2xd\" (UniqueName: \"kubernetes.io/projected/6ef16b68-fef4-40d7-8ca2-66b900480734-kube-api-access-pz2xd\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255091    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/325554ac-10f4-4076-b562-20b27ddf18b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4350004-c24b-49f4-9142-b7d268dd1132-host\") pod \"node-ca-wptfc\" (UID: \"f4350004-c24b-49f4-9142-b7d268dd1132\") " pod="openshift-image-registry/node-ca-wptfc"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255132    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/325554ac-10f4-4076-b562-20b27ddf18b0-system-cni-dir\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255184    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4350004-c24b-49f4-9142-b7d268dd1132-host\") pod \"node-ca-wptfc\" (UID: \"f4350004-c24b-49f4-9142-b7d268dd1132\") " pod="openshift-image-registry/node-ca-wptfc"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255188    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/325554ac-10f4-4076-b562-20b27ddf18b0-cnibin\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255210    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/325554ac-10f4-4076-b562-20b27ddf18b0-cnibin\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255225    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/325554ac-10f4-4076-b562-20b27ddf18b0-system-cni-dir\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255229    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-var-lib-openvswitch\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255280    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-sysctl-d\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.256199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255327    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnqqq\" (UniqueName: \"kubernetes.io/projected/2c67fffb-d71c-4672-9b51-d55117050dc2-kube-api-access-xnqqq\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255361    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae4f1d2d-d42f-445d-aa5a-88659cbf4598-iptables-alerter-script\") pod \"iptables-alerter-7hs9z\" (UID: \"ae4f1d2d-d42f-445d-aa5a-88659cbf4598\") " pod="openshift-network-operator/iptables-alerter-7hs9z"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255388    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae4f1d2d-d42f-445d-aa5a-88659cbf4598-host-slash\") pod \"iptables-alerter-7hs9z\" (UID: \"ae4f1d2d-d42f-445d-aa5a-88659cbf4598\") " pod="openshift-network-operator/iptables-alerter-7hs9z"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255414    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bac64312-87a3-4c2c-8cfa-80754d0123ed-hosts-file\") pod \"node-resolver-gcwqn\" (UID: \"bac64312-87a3-4c2c-8cfa-80754d0123ed\") " pod="openshift-dns/node-resolver-gcwqn"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255439    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5shf\" (UniqueName: \"kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf\") pod \"network-check-target-n6cbg\" (UID: \"e041de26-66e3-4855-9270-cf54fe1d8045\") " pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255466    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-multus-cni-dir\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255488    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae4f1d2d-d42f-445d-aa5a-88659cbf4598-host-slash\") pod \"iptables-alerter-7hs9z\" (UID: \"ae4f1d2d-d42f-445d-aa5a-88659cbf4598\") " pod="openshift-network-operator/iptables-alerter-7hs9z"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255492    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bac64312-87a3-4c2c-8cfa-80754d0123ed-hosts-file\") pod \"node-resolver-gcwqn\" (UID: \"bac64312-87a3-4c2c-8cfa-80754d0123ed\") " pod="openshift-dns/node-resolver-gcwqn"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255495    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42x4d\" (UniqueName: \"kubernetes.io/projected/beefa577-daaa-474f-9acf-80700f353213-kube-api-access-42x4d\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255549    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-run-openvswitch\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255575    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw284\" (UniqueName: \"kubernetes.io/projected/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-kube-api-access-fw284\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255603    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsnwf\" (UniqueName: \"kubernetes.io/projected/bac64312-87a3-4c2c-8cfa-80754d0123ed-kube-api-access-vsnwf\") pod \"node-resolver-gcwqn\" (UID: \"bac64312-87a3-4c2c-8cfa-80754d0123ed\") " pod="openshift-dns/node-resolver-gcwqn"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255613    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-multus-cni-dir\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255631    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-system-cni-dir\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255671    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-multus-socket-dir-parent\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255696    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-system-cni-dir\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255729    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/beefa577-daaa-474f-9acf-80700f353213-multus-daemon-config\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.257122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255768    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-multus-socket-dir-parent\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255811    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255877    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-modprobe-d\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255929    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-etc-selinux\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.255976    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkbw5\" (UniqueName: \"kubernetes.io/projected/ae4f1d2d-d42f-445d-aa5a-88659cbf4598-kube-api-access-jkbw5\") pod \"iptables-alerter-7hs9z\" (UID: \"ae4f1d2d-d42f-445d-aa5a-88659cbf4598\") " pod="openshift-network-operator/iptables-alerter-7hs9z"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256002    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-run-netns\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256029    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-ovnkube-script-lib\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256043    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae4f1d2d-d42f-445d-aa5a-88659cbf4598-iptables-alerter-script\") pod \"iptables-alerter-7hs9z\" (UID: \"ae4f1d2d-d42f-445d-aa5a-88659cbf4598\") " pod="openshift-network-operator/iptables-alerter-7hs9z"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256054    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-tuned\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs\") pod \"network-metrics-daemon-zqjzm\" (UID: \"0ee703af-bd7c-45e1-8b8a-1b31645d40bf\") " pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256151    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-run-k8s-cni-cncf-io\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256170    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-run-netns\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256187    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-var-lib-cni-multus\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256218    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-sysctl-conf\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256237    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-run-k8s-cni-cncf-io\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256261    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-run-netns\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.257736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256262    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-host\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256279    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/beefa577-daaa-474f-9acf-80700f353213-multus-daemon-config\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256280    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/beefa577-daaa-474f-9acf-80700f353213-host-var-lib-cni-multus\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256304    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-socket-dir\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256333    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-device-dir\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256366    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/325554ac-10f4-4076-b562-20b27ddf18b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256396    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-node-log\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256427    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2de9f57b-63bf-4dad-be1a-86716a7758f8-kubelet-config\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256453    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-sys\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4350004-c24b-49f4-9142-b7d268dd1132-serviceca\") pod \"node-ca-wptfc\" (UID: \"f4350004-c24b-49f4-9142-b7d268dd1132\") " pod="openshift-image-registry/node-ca-wptfc"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256507    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmg6v\" (UniqueName: \"kubernetes.io/projected/325554ac-10f4-4076-b562-20b27ddf18b0-kube-api-access-xmg6v\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256535    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256590    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256616    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-lib-modules\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256644    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/beefa577-daaa-474f-9acf-80700f353213-cni-binary-copy\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256868    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/325554ac-10f4-4076-b562-20b27ddf18b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.256902    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4350004-c24b-49f4-9142-b7d268dd1132-serviceca\") pod \"node-ca-wptfc\" (UID: \"f4350004-c24b-49f4-9142-b7d268dd1132\") " pod="openshift-image-registry/node-ca-wptfc"
Jul 28 17:16:45.258307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.257220    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/beefa577-daaa-474f-9acf-80700f353213-cni-binary-copy\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.258863 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.258341    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ab5d559-43a9-4c98-9576-48e592b09677-agent-certs\") pod \"konnectivity-agent-vvdjg\" (UID: \"1ab5d559-43a9-4c98-9576-48e592b09677\") " pod="kube-system/konnectivity-agent-vvdjg"
Jul 28 17:16:45.261525 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.261490    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv2vs\" (UniqueName: \"kubernetes.io/projected/f4350004-c24b-49f4-9142-b7d268dd1132-kube-api-access-rv2vs\") pod \"node-ca-wptfc\" (UID: \"f4350004-c24b-49f4-9142-b7d268dd1132\") " pod="openshift-image-registry/node-ca-wptfc"
Jul 28 17:16:45.262144 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.262121    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:16:45.262281 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.262149    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:16:45.262281 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.262186    2575 projected.go:196] Error preparing data for projected volume kube-api-access-l5shf for pod openshift-network-diagnostics/network-check-target-n6cbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:45.262281 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.262278    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf podName:e041de26-66e3-4855-9270-cf54fe1d8045 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:45.762240069 +0000 UTC m=+2.019849762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l5shf" (UniqueName: "kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf") pod "network-check-target-n6cbg" (UID: "e041de26-66e3-4855-9270-cf54fe1d8045") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:45.263858 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.263835    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42x4d\" (UniqueName: \"kubernetes.io/projected/beefa577-daaa-474f-9acf-80700f353213-kube-api-access-42x4d\") pod \"multus-l4s6b\" (UID: \"beefa577-daaa-474f-9acf-80700f353213\") " pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.263935 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.263916    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmg6v\" (UniqueName: \"kubernetes.io/projected/325554ac-10f4-4076-b562-20b27ddf18b0-kube-api-access-xmg6v\") pod \"multus-additional-cni-plugins-xnpnk\" (UID: \"325554ac-10f4-4076-b562-20b27ddf18b0\") " pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.264210 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.264195    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkbw5\" (UniqueName: \"kubernetes.io/projected/ae4f1d2d-d42f-445d-aa5a-88659cbf4598-kube-api-access-jkbw5\") pod \"iptables-alerter-7hs9z\" (UID: \"ae4f1d2d-d42f-445d-aa5a-88659cbf4598\") " pod="openshift-network-operator/iptables-alerter-7hs9z"
Jul 28 17:16:45.264516 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.264496    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsnwf\" (UniqueName: \"kubernetes.io/projected/bac64312-87a3-4c2c-8cfa-80754d0123ed-kube-api-access-vsnwf\") pod \"node-resolver-gcwqn\" (UID: \"bac64312-87a3-4c2c-8cfa-80754d0123ed\") " pod="openshift-dns/node-resolver-gcwqn"
Jul 28 17:16:45.308067 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:45.308021    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44dbed9217caa115cd6a389d801add6e.slice/crio-282c574fb1a8662a3869238921d3bb4fdb269d8f6ea822b05d491c9e69b1155e WatchSource:0}: Error finding container 282c574fb1a8662a3869238921d3bb4fdb269d8f6ea822b05d491c9e69b1155e: Status 404 returned error can't find the container with id 282c574fb1a8662a3869238921d3bb4fdb269d8f6ea822b05d491c9e69b1155e
Jul 28 17:16:45.311534 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.311517    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:16:45.357680 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357589    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz2xd\" (UniqueName: \"kubernetes.io/projected/6ef16b68-fef4-40d7-8ca2-66b900480734-kube-api-access-pz2xd\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.357680 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357629    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-var-lib-openvswitch\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.357680 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357650    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-sysctl-d\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.357680 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357679    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnqqq\" (UniqueName: \"kubernetes.io/projected/2c67fffb-d71c-4672-9b51-d55117050dc2-kube-api-access-xnqqq\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357702    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-var-lib-openvswitch\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357716    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-run-openvswitch\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357733    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw284\" (UniqueName: \"kubernetes.io/projected/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-kube-api-access-fw284\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357755    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357760    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-run-openvswitch\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357779    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-modprobe-d\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357801    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357814    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-sysctl-d\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357840    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-etc-selinux\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-run-netns\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357870    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357892    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-ovnkube-script-lib\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357917    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-tuned\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357916    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-modprobe-d\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357937    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs\") pod \"network-metrics-daemon-zqjzm\" (UID: \"0ee703af-bd7c-45e1-8b8a-1b31645d40bf\") " pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:45.358024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357943    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-run-netns\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357955    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-sysctl-conf\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358025    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-host\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.358039    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.357961    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-etc-selinux\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358059    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-socket-dir\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358108    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-host\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358106    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-device-dir\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358068    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-sysctl-conf\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.358142    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs podName:0ee703af-bd7c-45e1-8b8a-1b31645d40bf nodeName:}" failed. No retries permitted until 2026-07-28 17:16:45.8581173 +0000 UTC m=+2.115726982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs") pod "network-metrics-daemon-zqjzm" (UID: "0ee703af-bd7c-45e1-8b8a-1b31645d40bf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358174    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-socket-dir\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358191    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-node-log\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358177    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-device-dir\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358218    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2de9f57b-63bf-4dad-be1a-86716a7758f8-kubelet-config\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358239    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-node-log\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358246    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-sys\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358293    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.358903 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358317    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358355    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-lib-modules\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358369    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358380    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-run-ovn\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.358401    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358430    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-env-overrides\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.358453    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret podName:2de9f57b-63bf-4dad-be1a-86716a7758f8 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:45.85843657 +0000 UTC m=+2.116046244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret") pod "global-pull-secret-syncer-8sv5v" (UID: "2de9f57b-63bf-4dad-be1a-86716a7758f8") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358488    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-cni-bin\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358507    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-run-ovn\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-kubernetes\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-cni-netd\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358571    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-cni-bin\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358297    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2de9f57b-63bf-4dad-be1a-86716a7758f8-kubelet-config\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358491    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-lib-modules\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358614    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-ovnkube-script-lib\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358587    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-systemd-units\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358316    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-sys\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.359789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358651    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-kubernetes\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358654    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-run-systemd\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358679    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-cni-netd\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358696    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-ovnkube-config\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358696    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-systemd-units\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358700    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-run-systemd\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9qwq\" (UniqueName: \"kubernetes.io/projected/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-kube-api-access-l9qwq\") pod \"network-metrics-daemon-zqjzm\" (UID: \"0ee703af-bd7c-45e1-8b8a-1b31645d40bf\") " pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358760    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-slash\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358783    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-log-socket\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358822    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-log-socket\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358839    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-slash\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358849    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-env-overrides\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358901    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2de9f57b-63bf-4dad-be1a-86716a7758f8-dbus\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358932    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-systemd\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358960    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2de9f57b-63bf-4dad-be1a-86716a7758f8-dbus\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358959    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-etc-openvswitch\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.358996    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-systemd\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359002    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-run\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.360777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359004    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-etc-openvswitch\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359027    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ef16b68-fef4-40d7-8ca2-66b900480734-tmp\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359045    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-kubelet\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359062    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-run\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-sysconfig\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359099    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-sysconfig\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359103    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-registration-dir\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359108    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-host-kubelet\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359131    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-sys-fs\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359151    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-registration-dir\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359155    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-ovnkube-config\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359159    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-ovn-node-metrics-cert\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359196    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2c67fffb-d71c-4672-9b51-d55117050dc2-sys-fs\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359213    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-var-lib-kubelet\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.359267    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef16b68-fef4-40d7-8ca2-66b900480734-var-lib-kubelet\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.360878    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ef16b68-fef4-40d7-8ca2-66b900480734-etc-tuned\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.361142    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-ovn-node-metrics-cert\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.361383 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.361162    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ef16b68-fef4-40d7-8ca2-66b900480734-tmp\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.369381 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.369351    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnqqq\" (UniqueName: \"kubernetes.io/projected/2c67fffb-d71c-4672-9b51-d55117050dc2-kube-api-access-xnqqq\") pod \"aws-ebs-csi-driver-node-bdlx6\" (UID: \"2c67fffb-d71c-4672-9b51-d55117050dc2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.369577 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.369558    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz2xd\" (UniqueName: \"kubernetes.io/projected/6ef16b68-fef4-40d7-8ca2-66b900480734-kube-api-access-pz2xd\") pod \"tuned-8f9mp\" (UID: \"6ef16b68-fef4-40d7-8ca2-66b900480734\") " pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.369704 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.369687    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw284\" (UniqueName: \"kubernetes.io/projected/f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe-kube-api-access-fw284\") pod \"ovnkube-node-c954x\" (UID: \"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.369798 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.369782    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9qwq\" (UniqueName: \"kubernetes.io/projected/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-kube-api-access-l9qwq\") pod \"network-metrics-daemon-zqjzm\" (UID: \"0ee703af-bd7c-45e1-8b8a-1b31645d40bf\") " pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:45.461467 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.461427    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l4s6b"
Jul 28 17:16:45.469820 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.469794    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:45.474297 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:45.474269    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeefa577_daaa_474f_9acf_80700f353213.slice/crio-a3a70b1e67e76dee8fc8c1c9a4a249853bf18a1cc41f8d6113efe175178a9ffa WatchSource:0}: Error finding container a3a70b1e67e76dee8fc8c1c9a4a249853bf18a1cc41f8d6113efe175178a9ffa: Status 404 returned error can't find the container with id a3a70b1e67e76dee8fc8c1c9a4a249853bf18a1cc41f8d6113efe175178a9ffa
Jul 28 17:16:45.475023 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:45.474997    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11997b299c0cb2944eaf7c6feeeb8aab.slice/crio-ff08ce2dd71eeddd0a6d218094110e6bfe4f8edf7091fffa0bad4315bc4b4d07 WatchSource:0}: Error finding container ff08ce2dd71eeddd0a6d218094110e6bfe4f8edf7091fffa0bad4315bc4b4d07: Status 404 returned error can't find the container with id ff08ce2dd71eeddd0a6d218094110e6bfe4f8edf7091fffa0bad4315bc4b4d07
Jul 28 17:16:45.493699 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.493666    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7hs9z"
Jul 28 17:16:45.502035 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.502011    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gcwqn"
Jul 28 17:16:45.502181 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:45.502134    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4f1d2d_d42f_445d_aa5a_88659cbf4598.slice/crio-b2c11bbe485770980d59d975fdbc6eed36e3189e902a633fd439bc2b0a12f486 WatchSource:0}: Error finding container b2c11bbe485770980d59d975fdbc6eed36e3189e902a633fd439bc2b0a12f486: Status 404 returned error can't find the container with id b2c11bbe485770980d59d975fdbc6eed36e3189e902a633fd439bc2b0a12f486
Jul 28 17:16:45.510777 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:45.510754    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac64312_87a3_4c2c_8cfa_80754d0123ed.slice/crio-a8b2ffd28669b158a1a071660c23b7e6321ec5760c0cf098dac3914d4caf8316 WatchSource:0}: Error finding container a8b2ffd28669b158a1a071660c23b7e6321ec5760c0cf098dac3914d4caf8316: Status 404 returned error can't find the container with id a8b2ffd28669b158a1a071660c23b7e6321ec5760c0cf098dac3914d4caf8316
Jul 28 17:16:45.517023 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.517002    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vvdjg"
Jul 28 17:16:45.524703 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:45.524677    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab5d559_43a9_4c98_9576_48e592b09677.slice/crio-2172c48d1f62ff441955826f35564fc7b53ed0fdb7292246d6c95760b7272b60 WatchSource:0}: Error finding container 2172c48d1f62ff441955826f35564fc7b53ed0fdb7292246d6c95760b7272b60: Status 404 returned error can't find the container with id 2172c48d1f62ff441955826f35564fc7b53ed0fdb7292246d6c95760b7272b60
Jul 28 17:16:45.535749 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.535732    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wptfc"
Jul 28 17:16:45.543491 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:45.543467    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4350004_c24b_49f4_9142_b7d268dd1132.slice/crio-2a2f1a7b6f1b4b87e462b21e0704bdbf6cb4ff0c91970b403575d23524944da3 WatchSource:0}: Error finding container 2a2f1a7b6f1b4b87e462b21e0704bdbf6cb4ff0c91970b403575d23524944da3: Status 404 returned error can't find the container with id 2a2f1a7b6f1b4b87e462b21e0704bdbf6cb4ff0c91970b403575d23524944da3
Jul 28 17:16:45.546454 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.546436    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xnpnk"
Jul 28 17:16:45.554094 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:45.554068    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325554ac_10f4_4076_b562_20b27ddf18b0.slice/crio-8d25a1dcd846b214c2b6d6e87493c11193337f29451d33b50f403110b4cd51e3 WatchSource:0}: Error finding container 8d25a1dcd846b214c2b6d6e87493c11193337f29451d33b50f403110b4cd51e3: Status 404 returned error can't find the container with id 8d25a1dcd846b214c2b6d6e87493c11193337f29451d33b50f403110b4cd51e3
Jul 28 17:16:45.563680 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.563660    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:16:45.571395 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:45.571371    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e3a41d_925d_44b9_a6cb_ccf17cabd6fe.slice/crio-558407b07ddfbbc653565120ebe702a220aaf4f2a0b8d871e450d37b722a2454 WatchSource:0}: Error finding container 558407b07ddfbbc653565120ebe702a220aaf4f2a0b8d871e450d37b722a2454: Status 404 returned error can't find the container with id 558407b07ddfbbc653565120ebe702a220aaf4f2a0b8d871e450d37b722a2454
Jul 28 17:16:45.577721 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.577699    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8f9mp"
Jul 28 17:16:45.581896 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.581880    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6"
Jul 28 17:16:45.585076 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:45.585052    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef16b68_fef4_40d7_8ca2_66b900480734.slice/crio-c359bcfd9bd7ecb7f93e4264b2d9ce2d5e381c3a6f2050e46c31f8270cd8af1f WatchSource:0}: Error finding container c359bcfd9bd7ecb7f93e4264b2d9ce2d5e381c3a6f2050e46c31f8270cd8af1f: Status 404 returned error can't find the container with id c359bcfd9bd7ecb7f93e4264b2d9ce2d5e381c3a6f2050e46c31f8270cd8af1f
Jul 28 17:16:45.590072 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:16:45.590050    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c67fffb_d71c_4672_9b51_d55117050dc2.slice/crio-61b17cc848ab0f70dfedd84b10070a24110b259762caf4990a0eb8ca171a1858 WatchSource:0}: Error finding container 61b17cc848ab0f70dfedd84b10070a24110b259762caf4990a0eb8ca171a1858: Status 404 returned error can't find the container with id 61b17cc848ab0f70dfedd84b10070a24110b259762caf4990a0eb8ca171a1858
Jul 28 17:16:45.763139 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.763054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5shf\" (UniqueName: \"kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf\") pod \"network-check-target-n6cbg\" (UID: \"e041de26-66e3-4855-9270-cf54fe1d8045\") " pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:16:45.763280 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.763193    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:16:45.763280 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.763210    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:16:45.763280 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.763220    2575 projected.go:196] Error preparing data for projected volume kube-api-access-l5shf for pod openshift-network-diagnostics/network-check-target-n6cbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:45.763280 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.763271    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf podName:e041de26-66e3-4855-9270-cf54fe1d8045 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:46.763258019 +0000 UTC m=+3.020867688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5shf" (UniqueName: "kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf") pod "network-check-target-n6cbg" (UID: "e041de26-66e3-4855-9270-cf54fe1d8045") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:45.864404 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.864371    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs\") pod \"network-metrics-daemon-zqjzm\" (UID: \"0ee703af-bd7c-45e1-8b8a-1b31645d40bf\") " pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:45.864597 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:45.864413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:45.864597 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.864506    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:45.864597 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.864542    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:45.864597 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.864589    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret podName:2de9f57b-63bf-4dad-be1a-86716a7758f8 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:46.864574932 +0000 UTC m=+3.122184601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret") pod "global-pull-secret-syncer-8sv5v" (UID: "2de9f57b-63bf-4dad-be1a-86716a7758f8") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:45.864798 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:45.864603    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs podName:0ee703af-bd7c-45e1-8b8a-1b31645d40bf nodeName:}" failed. No retries permitted until 2026-07-28 17:16:46.864596928 +0000 UTC m=+3.122206597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs") pod "network-metrics-daemon-zqjzm" (UID: "0ee703af-bd7c-45e1-8b8a-1b31645d40bf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:46.199503 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.199413    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:11:45 +0000 UTC" deadline="2028-05-12 03:53:16.092662312 +0000 UTC"
Jul 28 17:16:46.199503 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.199452    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15682h36m29.893214828s"
Jul 28 17:16:46.201726 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.201676    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:46.308637 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.308590    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:16:46.308855 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:46.308740    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:16:46.309272 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.309245    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:46.309397 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:46.309366    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:16:46.327392 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.327316    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal" event={"ID":"44dbed9217caa115cd6a389d801add6e","Type":"ContainerStarted","Data":"282c574fb1a8662a3869238921d3bb4fdb269d8f6ea822b05d491c9e69b1155e"}
Jul 28 17:16:46.332827 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.332798    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6" event={"ID":"2c67fffb-d71c-4672-9b51-d55117050dc2","Type":"ContainerStarted","Data":"61b17cc848ab0f70dfedd84b10070a24110b259762caf4990a0eb8ca171a1858"}
Jul 28 17:16:46.341977 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.341950    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnpnk" event={"ID":"325554ac-10f4-4076-b562-20b27ddf18b0","Type":"ContainerStarted","Data":"8d25a1dcd846b214c2b6d6e87493c11193337f29451d33b50f403110b4cd51e3"}
Jul 28 17:16:46.345290 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.345265    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wptfc" event={"ID":"f4350004-c24b-49f4-9142-b7d268dd1132","Type":"ContainerStarted","Data":"2a2f1a7b6f1b4b87e462b21e0704bdbf6cb4ff0c91970b403575d23524944da3"}
Jul 28 17:16:46.351558 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.351533    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vvdjg" event={"ID":"1ab5d559-43a9-4c98-9576-48e592b09677","Type":"ContainerStarted","Data":"2172c48d1f62ff441955826f35564fc7b53ed0fdb7292246d6c95760b7272b60"}
Jul 28 17:16:46.360415 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.360266    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gcwqn" event={"ID":"bac64312-87a3-4c2c-8cfa-80754d0123ed","Type":"ContainerStarted","Data":"a8b2ffd28669b158a1a071660c23b7e6321ec5760c0cf098dac3914d4caf8316"}
Jul 28 17:16:46.373522 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.373495    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7hs9z" event={"ID":"ae4f1d2d-d42f-445d-aa5a-88659cbf4598","Type":"ContainerStarted","Data":"b2c11bbe485770980d59d975fdbc6eed36e3189e902a633fd439bc2b0a12f486"}
Jul 28 17:16:46.393624 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.391116    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal" event={"ID":"11997b299c0cb2944eaf7c6feeeb8aab","Type":"ContainerStarted","Data":"ff08ce2dd71eeddd0a6d218094110e6bfe4f8edf7091fffa0bad4315bc4b4d07"}
Jul 28 17:16:46.410564 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.410478    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l4s6b" event={"ID":"beefa577-daaa-474f-9acf-80700f353213","Type":"ContainerStarted","Data":"a3a70b1e67e76dee8fc8c1c9a4a249853bf18a1cc41f8d6113efe175178a9ffa"}
Jul 28 17:16:46.413601 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.413506    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8f9mp" event={"ID":"6ef16b68-fef4-40d7-8ca2-66b900480734","Type":"ContainerStarted","Data":"c359bcfd9bd7ecb7f93e4264b2d9ce2d5e381c3a6f2050e46c31f8270cd8af1f"}
Jul 28 17:16:46.432874 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.432839    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c954x" event={"ID":"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe","Type":"ContainerStarted","Data":"558407b07ddfbbc653565120ebe702a220aaf4f2a0b8d871e450d37b722a2454"}
Jul 28 17:16:46.773825 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.773764    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5shf\" (UniqueName: \"kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf\") pod \"network-check-target-n6cbg\" (UID: \"e041de26-66e3-4855-9270-cf54fe1d8045\") " pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:16:46.774013 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:46.773958    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:16:46.774013 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:46.773985    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:16:46.774013 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:46.773997    2575 projected.go:196] Error preparing data for projected volume kube-api-access-l5shf for pod openshift-network-diagnostics/network-check-target-n6cbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:46.774201 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:46.774060    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf podName:e041de26-66e3-4855-9270-cf54fe1d8045 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:48.774041505 +0000 UTC m=+5.031651177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5shf" (UniqueName: "kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf") pod "network-check-target-n6cbg" (UID: "e041de26-66e3-4855-9270-cf54fe1d8045") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:46.876700 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.875918    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs\") pod \"network-metrics-daemon-zqjzm\" (UID: \"0ee703af-bd7c-45e1-8b8a-1b31645d40bf\") " pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:46.876700 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:46.875977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:46.876700 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:46.876132    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:46.876700 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:46.876196    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret podName:2de9f57b-63bf-4dad-be1a-86716a7758f8 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:48.876177078 +0000 UTC m=+5.133786751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret") pod "global-pull-secret-syncer-8sv5v" (UID: "2de9f57b-63bf-4dad-be1a-86716a7758f8") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:46.876700 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:46.876611    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:46.876700 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:46.876667    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs podName:0ee703af-bd7c-45e1-8b8a-1b31645d40bf nodeName:}" failed. No retries permitted until 2026-07-28 17:16:48.876650184 +0000 UTC m=+5.134259860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs") pod "network-metrics-daemon-zqjzm" (UID: "0ee703af-bd7c-45e1-8b8a-1b31645d40bf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:47.200689 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:47.200575    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:11:45 +0000 UTC" deadline="2028-02-12 04:08:28.789111332 +0000 UTC"
Jul 28 17:16:47.200689 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:47.200638    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13522h51m41.588477925s"
Jul 28 17:16:47.306615 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:47.306568    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:47.306847 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:47.306710    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:16:47.574667 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:47.574555    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:48.280271 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:48.280242    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:48.308872 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:48.308828    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:16:48.309052 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:48.308961    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:16:48.309578 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:48.309370    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:48.309578 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:48.309487    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:16:48.792690 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:48.792652    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5shf\" (UniqueName: \"kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf\") pod \"network-check-target-n6cbg\" (UID: \"e041de26-66e3-4855-9270-cf54fe1d8045\") " pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:16:48.792899 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:48.792880    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:16:48.792989 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:48.792908    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:16:48.792989 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:48.792922    2575 projected.go:196] Error preparing data for projected volume kube-api-access-l5shf for pod openshift-network-diagnostics/network-check-target-n6cbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:48.792989 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:48.792985    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf podName:e041de26-66e3-4855-9270-cf54fe1d8045 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:52.792964932 +0000 UTC m=+9.050574623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5shf" (UniqueName: "kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf") pod "network-check-target-n6cbg" (UID: "e041de26-66e3-4855-9270-cf54fe1d8045") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:48.893678 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:48.893628    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs\") pod \"network-metrics-daemon-zqjzm\" (UID: \"0ee703af-bd7c-45e1-8b8a-1b31645d40bf\") " pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:48.893873 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:48.893691    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:48.893873 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:48.893831    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:48.893992 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:48.893897    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret podName:2de9f57b-63bf-4dad-be1a-86716a7758f8 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:52.893876933 +0000 UTC m=+9.151486612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret") pod "global-pull-secret-syncer-8sv5v" (UID: "2de9f57b-63bf-4dad-be1a-86716a7758f8") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:48.893992 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:48.893832    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:48.894107 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:48.893984    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs podName:0ee703af-bd7c-45e1-8b8a-1b31645d40bf nodeName:}" failed. No retries permitted until 2026-07-28 17:16:52.893965668 +0000 UTC m=+9.151575342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs") pod "network-metrics-daemon-zqjzm" (UID: "0ee703af-bd7c-45e1-8b8a-1b31645d40bf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:49.306882 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:49.306842    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:49.307360 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:49.306990    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:16:50.306699 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:50.306664    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:16:50.306884 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:50.306799    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:16:50.308281 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:50.308252    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:50.308744 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:50.308393    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:16:51.306733 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:51.306697    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:51.306928 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:51.306841    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:16:52.306080 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:52.306048    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:16:52.306560 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:52.306051    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:52.306560 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:52.306189    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:16:52.306560 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:52.306282    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:16:52.832933 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:52.832274    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5shf\" (UniqueName: \"kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf\") pod \"network-check-target-n6cbg\" (UID: \"e041de26-66e3-4855-9270-cf54fe1d8045\") " pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:16:52.832933 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:52.832487    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:16:52.832933 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:52.832507    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:16:52.832933 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:52.832535    2575 projected.go:196] Error preparing data for projected volume kube-api-access-l5shf for pod openshift-network-diagnostics/network-check-target-n6cbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:52.832933 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:52.832592    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf podName:e041de26-66e3-4855-9270-cf54fe1d8045 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:00.832575457 +0000 UTC m=+17.090185129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5shf" (UniqueName: "kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf") pod "network-check-target-n6cbg" (UID: "e041de26-66e3-4855-9270-cf54fe1d8045") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:52.933978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:52.933463    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs\") pod \"network-metrics-daemon-zqjzm\" (UID: \"0ee703af-bd7c-45e1-8b8a-1b31645d40bf\") " pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:52.933978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:52.933539    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:52.933978 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:52.933700    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:52.933978 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:52.933762    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret podName:2de9f57b-63bf-4dad-be1a-86716a7758f8 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:00.93374369 +0000 UTC m=+17.191353360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret") pod "global-pull-secret-syncer-8sv5v" (UID: "2de9f57b-63bf-4dad-be1a-86716a7758f8") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:52.933978 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:52.933832    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:52.933978 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:52.933871    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs podName:0ee703af-bd7c-45e1-8b8a-1b31645d40bf nodeName:}" failed. No retries permitted until 2026-07-28 17:17:00.933856225 +0000 UTC m=+17.191465892 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs") pod "network-metrics-daemon-zqjzm" (UID: "0ee703af-bd7c-45e1-8b8a-1b31645d40bf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:53.306568 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:53.306515    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:53.307056 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:53.306672    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:16:54.308259 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:54.307630    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:16:54.308259 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:54.307741    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:16:54.308259 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:54.308111    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:54.308259 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:54.308211    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:16:55.306323 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:55.306283    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:55.306498 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:55.306433    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:16:56.306743 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:56.306701    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:16:56.306743 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:56.306736    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:56.307238 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:56.306849    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:16:56.307238 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:56.307008    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:16:57.305833 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:57.305800    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:57.306016 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:57.305926    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:16:58.306593 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:58.306481    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:16:58.306593 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:58.306538    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:16:58.307130 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:58.306654    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:16:58.307130 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:58.306772    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:16:59.306677 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:16:59.306644    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:16:59.307068 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:16:59.306769    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:17:00.305893 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:00.305862    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:17:00.305893 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:00.305897    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:00.306141 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:00.305990    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:17:00.306141 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:00.306094    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:17:00.893488 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:00.893446    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5shf\" (UniqueName: \"kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf\") pod \"network-check-target-n6cbg\" (UID: \"e041de26-66e3-4855-9270-cf54fe1d8045\") " pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:00.893915 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:00.893626    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:00.893915 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:00.893645    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:00.893915 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:00.893654    2575 projected.go:196] Error preparing data for projected volume kube-api-access-l5shf for pod openshift-network-diagnostics/network-check-target-n6cbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:00.893915 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:00.893712    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf podName:e041de26-66e3-4855-9270-cf54fe1d8045 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:16.893696752 +0000 UTC m=+33.151306439 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5shf" (UniqueName: "kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf") pod "network-check-target-n6cbg" (UID: "e041de26-66e3-4855-9270-cf54fe1d8045") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:00.994865 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:00.994833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs\") pod \"network-metrics-daemon-zqjzm\" (UID: \"0ee703af-bd7c-45e1-8b8a-1b31645d40bf\") " pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:17:00.995025 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:00.994885    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:00.995025 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:00.994992    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:00.995093 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:00.995051    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret podName:2de9f57b-63bf-4dad-be1a-86716a7758f8 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:16.995033857 +0000 UTC m=+33.252643526 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret") pod "global-pull-secret-syncer-8sv5v" (UID: "2de9f57b-63bf-4dad-be1a-86716a7758f8") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:00.995093 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:00.994992    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:00.995159 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:00.995136    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs podName:0ee703af-bd7c-45e1-8b8a-1b31645d40bf nodeName:}" failed. No retries permitted until 2026-07-28 17:17:16.995121525 +0000 UTC m=+33.252731193 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs") pod "network-metrics-daemon-zqjzm" (UID: "0ee703af-bd7c-45e1-8b8a-1b31645d40bf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:01.306394 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:01.306372    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:01.306496 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:01.306471    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:17:02.306674 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:02.306636    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:02.306674 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:02.306657    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:17:02.307158 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:02.306771    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:17:02.307158 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:02.306895    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:17:03.306423 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:03.306384    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:03.306618 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:03.306542    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:17:04.306883 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:04.306844    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:17:04.307339 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:04.306957    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:17:04.307339 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:04.307011    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:04.307339 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:04.307135    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:17:05.306217 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:05.306189    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:05.306369 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:05.306286    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:17:06.306486 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.306327    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:06.307088 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.306394    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:17:06.307088 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:06.306578    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:17:06.307088 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:06.306668    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:17:06.479378 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.479348    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal" event={"ID":"11997b299c0cb2944eaf7c6feeeb8aab","Type":"ContainerStarted","Data":"8d1b7216d204d832ae8fc9a6a0d92b55b6f7f7e7abf6a72c83a850df9693c40c"}
Jul 28 17:17:06.481181 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.481155    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l4s6b" event={"ID":"beefa577-daaa-474f-9acf-80700f353213","Type":"ContainerStarted","Data":"3c091bad8a4283235e38d42da55371bdd46f5bf0dfe8d1a4bd5580d9602c0be8"}
Jul 28 17:17:06.482386 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.482364    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8f9mp" event={"ID":"6ef16b68-fef4-40d7-8ca2-66b900480734","Type":"ContainerStarted","Data":"909c961628ae21b75f60becd1c5400a699784153464ccc7503b858cac389525f"}
Jul 28 17:17:06.485024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.485007    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c954x" event={"ID":"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe","Type":"ContainerStarted","Data":"95686f0baae24a861fa09bc51dfbec8eb6c91fe52e10f7984a97960a38dfcbf1"}
Jul 28 17:17:06.485116 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.485063    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c954x" event={"ID":"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe","Type":"ContainerStarted","Data":"bb63bda96542475701b536c822fa5c87a5479f557606b8190a92bd4338feecfc"}
Jul 28 17:17:06.485116 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.485095    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c954x" event={"ID":"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe","Type":"ContainerStarted","Data":"17ce6b1c607c19c965fce8bdaf65676743f319bb82ada71b0f514ed8c24a1d54"}
Jul 28 17:17:06.485197 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.485126    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c954x" event={"ID":"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe","Type":"ContainerStarted","Data":"4df022856cf00aae9a05db607ffe2143c3f2a85d3f8ee6add6ccb8ae94ce553f"}
Jul 28 17:17:06.485197 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.485161    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c954x" event={"ID":"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe","Type":"ContainerStarted","Data":"e4ebc39b5d6f5ffbc7d2634c93768a63f7d5ce5e9be121b577f36a4a0105cc8a"}
Jul 28 17:17:06.485264 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.485193    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c954x" event={"ID":"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe","Type":"ContainerStarted","Data":"0a807e0c98ff2a9047c337a76b7f8f5b878eb1a8ec36b6998c7f765effd09b4b"}
Jul 28 17:17:06.486388 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.486365    2575 generic.go:356] "Generic (PLEG): container finished" podID="44dbed9217caa115cd6a389d801add6e" containerID="ae1d22f6ca277056ead2cfb7c01e81ed4bcb93af7811086693a05ebc0ee7543f" exitCode=0
Jul 28 17:17:06.486466 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.486433    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal" event={"ID":"44dbed9217caa115cd6a389d801add6e","Type":"ContainerDied","Data":"ae1d22f6ca277056ead2cfb7c01e81ed4bcb93af7811086693a05ebc0ee7543f"}
Jul 28 17:17:06.487784 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.487765    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6" event={"ID":"2c67fffb-d71c-4672-9b51-d55117050dc2","Type":"ContainerStarted","Data":"001e80933e96ee483444e2efd835b8c137cbf99603acf6f78060c3daa4fd02ad"}
Jul 28 17:17:06.489139 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.489120    2575 generic.go:356] "Generic (PLEG): container finished" podID="325554ac-10f4-4076-b562-20b27ddf18b0" containerID="3b4b5eaced40283a6be9830ebc1d3de7f6ba938deae9072655830e6740968c63" exitCode=0
Jul 28 17:17:06.489222 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.489142    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnpnk" event={"ID":"325554ac-10f4-4076-b562-20b27ddf18b0","Type":"ContainerDied","Data":"3b4b5eaced40283a6be9830ebc1d3de7f6ba938deae9072655830e6740968c63"}
Jul 28 17:17:06.490516 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.490498    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wptfc" event={"ID":"f4350004-c24b-49f4-9142-b7d268dd1132","Type":"ContainerStarted","Data":"074b705cb0fc99325e4242a6f218670e927dc99a69815913d8d9fb6b84b5c4aa"}
Jul 28 17:17:06.491796 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.491780    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vvdjg" event={"ID":"1ab5d559-43a9-4c98-9576-48e592b09677","Type":"ContainerStarted","Data":"ef297f64960b3ca3160c682626e4989b104cd13af4e97340f6ed71ff464e381b"}
Jul 28 17:17:06.492928 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.492871    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-188.ec2.internal" podStartSLOduration=21.49285924 podStartE2EDuration="21.49285924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:06.492829185 +0000 UTC m=+22.750438875" watchObservedRunningTime="2026-07-28 17:17:06.49285924 +0000 UTC m=+22.750468930"
Jul 28 17:17:06.493033 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.492972    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gcwqn" event={"ID":"bac64312-87a3-4c2c-8cfa-80754d0123ed","Type":"ContainerStarted","Data":"a07d188aa45475e682cb2e1aa1ed1fd1a195a4d474b1bd8a1ea62bffcee3f5c2"}
Jul 28 17:17:06.509376 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.509331    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l4s6b" podStartSLOduration=2.3942226189999998 podStartE2EDuration="22.509319776s" totalImagesPullingTime="20.115097157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:06.509127032 +0000 UTC m=+22.766736723" watchObservedRunningTime="2026-07-28 17:17:06.509319776 +0000 UTC m=+22.766929465"
Jul 28 17:17:06.527582 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.527540    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vvdjg" podStartSLOduration=2.661614214 podStartE2EDuration="22.527505996s" totalImagesPullingTime="19.865891782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:06.527365213 +0000 UTC m=+22.784974902" watchObservedRunningTime="2026-07-28 17:17:06.527505996 +0000 UTC m=+22.785115685"
Jul 28 17:17:06.542307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.542243    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wptfc" podStartSLOduration=1.697228676 podStartE2EDuration="21.542228647s" totalImagesPullingTime="19.844999971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:06.541891338 +0000 UTC m=+22.799501024" watchObservedRunningTime="2026-07-28 17:17:06.542228647 +0000 UTC m=+22.799838341"
Jul 28 17:17:06.590562 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.590497    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8f9mp" podStartSLOduration=1.5855112679999999 podStartE2EDuration="21.590482298s" totalImagesPullingTime="20.00497103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:06.589963786 +0000 UTC m=+22.847573476" watchObservedRunningTime="2026-07-28 17:17:06.590482298 +0000 UTC m=+22.848092018"
Jul 28 17:17:06.619916 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:06.619866    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gcwqn" podStartSLOduration=2.718795261 podStartE2EDuration="22.619843663s" totalImagesPullingTime="19.901048402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:06.619651691 +0000 UTC m=+22.877261389" watchObservedRunningTime="2026-07-28 17:17:06.619843663 +0000 UTC m=+22.877453354"
Jul 28 17:17:07.305944 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:07.305875    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:07.306125 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:07.305980    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:17:07.436824 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:07.436800    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:17:07.497757 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:07.497726    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6" event={"ID":"2c67fffb-d71c-4672-9b51-d55117050dc2","Type":"ContainerStarted","Data":"84c809d4f17e50cdd8b07310d26a9ce94034723115bc652cede1a2fec96a1b98"}
Jul 28 17:17:07.499283 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:07.499254    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7hs9z" event={"ID":"ae4f1d2d-d42f-445d-aa5a-88659cbf4598","Type":"ContainerStarted","Data":"394fa6801ec6ed8f7fdfe473a46e12aceccf1c8c53c02d0ec651de6b136d1be6"}
Jul 28 17:17:07.501071 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:07.501047    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal" event={"ID":"44dbed9217caa115cd6a389d801add6e","Type":"ContainerStarted","Data":"116bc442cb52a6f270d131174578026e00a473c29716bff4a8fa7c7f215b887a"}
Jul 28 17:17:07.514705 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:07.514664    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7hs9z" podStartSLOduration=3.62653897 podStartE2EDuration="23.514650918s" totalImagesPullingTime="19.888111948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:07.514247443 +0000 UTC m=+23.771857134" watchObservedRunningTime="2026-07-28 17:17:07.514650918 +0000 UTC m=+23.772260608"
Jul 28 17:17:07.529813 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:07.529770    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-188.ec2.internal" podStartSLOduration=22.529753604 podStartE2EDuration="22.529753604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:07.529532466 +0000 UTC m=+23.787142147" watchObservedRunningTime="2026-07-28 17:17:07.529753604 +0000 UTC m=+23.787363295"
Jul 28 17:17:08.216196 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:08.216100    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:17:07.436818691Z","UUID":"e929e1ce-6195-4595-8516-e393f5b3207f","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:17:08.218251 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:08.218230    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:17:08.218366 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:08.218268    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:17:08.306272 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:08.306240    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:17:08.306432 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:08.306358    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:17:08.306830 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:08.306805    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:08.307020 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:08.306906    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:17:09.306780 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:09.306723    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:09.307259 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:09.306857    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:17:09.509032 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:09.508974    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6" event={"ID":"2c67fffb-d71c-4672-9b51-d55117050dc2","Type":"ContainerStarted","Data":"fc9b1029ba464e85fdff50d1e6fd5615295f7491be8a80faa801fee73433f26b"}
Jul 28 17:17:09.512654 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:09.512624    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c954x" event={"ID":"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe","Type":"ContainerStarted","Data":"51caa8ebf71d8f8c966e85b30d30808a4b19be08021b56a3e9204e142b262363"}
Jul 28 17:17:09.526643 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:09.526595    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bdlx6" podStartSLOduration=2.264646835 podStartE2EDuration="24.526580044s" totalImagesPullingTime="22.261933209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:09.525901906 +0000 UTC m=+25.783511590" watchObservedRunningTime="2026-07-28 17:17:09.526580044 +0000 UTC m=+25.784189733"
Jul 28 17:17:10.306275 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:10.306239    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:10.306463 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:10.306372    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:17:10.306463 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:10.306429    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:17:10.306615 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:10.306557    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:17:11.306663 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:11.306456    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:11.307249 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:11.306751    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:17:11.425691 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:11.425661    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vvdjg"
Jul 28 17:17:11.426233 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:11.426208    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vvdjg"
Jul 28 17:17:11.520426 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:11.520391    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c954x" event={"ID":"f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe","Type":"ContainerStarted","Data":"bde57c419b00b24c6ca18226245d502fb67cdf8cbf21d61f5c3bc81bac17ecc7"}
Jul 28 17:17:11.520653 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:11.520632    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:17:11.520728 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:11.520707    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:17:11.522331 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:11.522297    2575 generic.go:356] "Generic (PLEG): container finished" podID="325554ac-10f4-4076-b562-20b27ddf18b0" containerID="9dc1e53e2121f8216a0807c9594394c9fe78c0125c3823a2ffb8213bb2f0ec17" exitCode=0
Jul 28 17:17:11.522417 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:11.522381    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnpnk" event={"ID":"325554ac-10f4-4076-b562-20b27ddf18b0","Type":"ContainerDied","Data":"9dc1e53e2121f8216a0807c9594394c9fe78c0125c3823a2ffb8213bb2f0ec17"}
Jul 28 17:17:11.522531 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:11.522503    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vvdjg"
Jul 28 17:17:11.523020 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:11.522955    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vvdjg"
Jul 28 17:17:11.537434 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:11.537409    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:17:11.559106 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:11.559069    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c954x" podStartSLOduration=6.422881192 podStartE2EDuration="26.559058469s" totalImagesPullingTime="20.136177277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:11.558659129 +0000 UTC m=+27.816268819" watchObservedRunningTime="2026-07-28 17:17:11.559058469 +0000 UTC m=+27.816668155"
Jul 28 17:17:12.306318 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:12.306278    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:17:12.306572 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:12.306294    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:12.306572 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:12.306433    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:17:12.306572 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:12.306492    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:17:12.525337 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:12.525309    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:17:12.539821 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:12.539793    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:17:12.933495 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:12.933280    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8sv5v"]
Jul 28 17:17:12.933695 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:12.933626    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:12.933757 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:12.933737    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:17:12.942780 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:12.942752    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zqjzm"]
Jul 28 17:17:12.942937 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:12.942921    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:17:12.943082 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:12.943048    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:17:12.950256 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:12.950233    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n6cbg"]
Jul 28 17:17:12.950354 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:12.950321    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:12.950415 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:12.950392    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:17:13.529166 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:13.529137    2575 generic.go:356] "Generic (PLEG): container finished" podID="325554ac-10f4-4076-b562-20b27ddf18b0" containerID="d287056e0f7ba7197806b628486cfb9f4cf8d5de01dbbff9a66b138a572e706a" exitCode=0
Jul 28 17:17:13.529592 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:13.529229    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnpnk" event={"ID":"325554ac-10f4-4076-b562-20b27ddf18b0","Type":"ContainerDied","Data":"d287056e0f7ba7197806b628486cfb9f4cf8d5de01dbbff9a66b138a572e706a"}
Jul 28 17:17:14.306632 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:14.306603    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:14.306768 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:14.306702    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:17:14.306819 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:14.306770    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:17:14.306883 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:14.306867    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:17:15.306843 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:15.306814    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:15.307261 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:15.306916    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:17:15.535625 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:15.535591    2575 generic.go:356] "Generic (PLEG): container finished" podID="325554ac-10f4-4076-b562-20b27ddf18b0" containerID="a4ddfc4217810a6ee6c42867bbd901164d28f3f24afea070bde28b75c83148b4" exitCode=0
Jul 28 17:17:15.535797 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:15.535634    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnpnk" event={"ID":"325554ac-10f4-4076-b562-20b27ddf18b0","Type":"ContainerDied","Data":"a4ddfc4217810a6ee6c42867bbd901164d28f3f24afea070bde28b75c83148b4"}
Jul 28 17:17:16.306270 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:16.306238    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:16.306490 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:16.306238    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:17:16.306490 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:16.306365    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:17:16.306490 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:16.306471    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:17:16.911504 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:16.911463    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5shf\" (UniqueName: \"kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf\") pod \"network-check-target-n6cbg\" (UID: \"e041de26-66e3-4855-9270-cf54fe1d8045\") " pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:16.911967 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:16.911668    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:16.911967 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:16.911690    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:16.911967 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:16.911700    2575 projected.go:196] Error preparing data for projected volume kube-api-access-l5shf for pod openshift-network-diagnostics/network-check-target-n6cbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:16.911967 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:16.911752    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf podName:e041de26-66e3-4855-9270-cf54fe1d8045 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:48.911736938 +0000 UTC m=+65.169346610 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5shf" (UniqueName: "kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf") pod "network-check-target-n6cbg" (UID: "e041de26-66e3-4855-9270-cf54fe1d8045") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:17.012829 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:17.012790    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs\") pod \"network-metrics-daemon-zqjzm\" (UID: \"0ee703af-bd7c-45e1-8b8a-1b31645d40bf\") " pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:17:17.012990 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:17.012846    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:17.012990 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:17.012943    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:17.013082 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:17.013007    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret podName:2de9f57b-63bf-4dad-be1a-86716a7758f8 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:49.012992706 +0000 UTC m=+65.270602375 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret") pod "global-pull-secret-syncer-8sv5v" (UID: "2de9f57b-63bf-4dad-be1a-86716a7758f8") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:17.013082 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:17.012948    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:17.013176 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:17.013107    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs podName:0ee703af-bd7c-45e1-8b8a-1b31645d40bf nodeName:}" failed. No retries permitted until 2026-07-28 17:17:49.013088359 +0000 UTC m=+65.270698046 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs") pod "network-metrics-daemon-zqjzm" (UID: "0ee703af-bd7c-45e1-8b8a-1b31645d40bf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:17.306893 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:17.306823    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:17.307038 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:17.306930    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:17:18.306138 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:18.306102    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:17:18.306138 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:18.306135    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:18.306690 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:18.306272    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqjzm" podUID="0ee703af-bd7c-45e1-8b8a-1b31645d40bf"
Jul 28 17:17:18.306690 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:18.306411    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv5v" podUID="2de9f57b-63bf-4dad-be1a-86716a7758f8"
Jul 28 17:17:19.306236 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.306201    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:19.306683 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:19.306334    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6cbg" podUID="e041de26-66e3-4855-9270-cf54fe1d8045"
Jul 28 17:17:19.591873 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.591851    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-188.ec2.internal" event="NodeReady"
Jul 28 17:17:19.591991 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.591977    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:17:19.631559 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.631374    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-dfbf9859d-q5c99"]
Jul 28 17:17:19.633917 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.633900    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.637201 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.637181    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 17:17:19.637299 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.637182    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 17:17:19.637299 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.637271    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 17:17:19.637711 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.637677    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gt8nm\""
Jul 28 17:17:19.643915 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.643893    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 17:17:19.656800 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.656416    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dfbf9859d-q5c99"]
Jul 28 17:17:19.657967 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.657945    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2nw7c"]
Jul 28 17:17:19.659897 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.659876    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5xnh4"]
Jul 28 17:17:19.660304 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.660284    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:19.661963 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.661944    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5xnh4"
Jul 28 17:17:19.663408 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.663384    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:17:19.663533 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.663505    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:17:19.663660 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.663644    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c9wr5\""
Jul 28 17:17:19.664101 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.664080    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:17:19.664194 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.664114    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dzv2b\""
Jul 28 17:17:19.664194 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.664170    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:17:19.664714 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.664700    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:17:19.670634 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.670610    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5xnh4"]
Jul 28 17:17:19.676468 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.676312    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2nw7c"]
Jul 28 17:17:19.733265 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.733238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-certificates\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.733363 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.733310    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a15ef051-198a-49eb-b21d-1d8a7b82ccba-ca-trust-extracted\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.733363 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.733336    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a15ef051-198a-49eb-b21d-1d8a7b82ccba-installation-pull-secrets\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.733363 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.733358    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8bp9\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-kube-api-access-s8bp9\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.733475 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.733440    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a15ef051-198a-49eb-b21d-1d8a7b82ccba-trusted-ca\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.733475 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.733466    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-bound-sa-token\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.733577 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.733497    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a15ef051-198a-49eb-b21d-1d8a7b82ccba-image-registry-private-configuration\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.733577 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.733548    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.834625 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.834538    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hmr7\" (UniqueName: \"kubernetes.io/projected/8bff935c-a6ff-41ec-8b79-8146ede38416-kube-api-access-9hmr7\") pod \"ingress-canary-5xnh4\" (UID: \"8bff935c-a6ff-41ec-8b79-8146ede38416\") " pod="openshift-ingress-canary/ingress-canary-5xnh4"
Jul 28 17:17:19.834625 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.834583    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-certificates\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.834830 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.834678    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a15ef051-198a-49eb-b21d-1d8a7b82ccba-ca-trust-extracted\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.834830 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.834709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a15ef051-198a-49eb-b21d-1d8a7b82ccba-installation-pull-secrets\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.834830 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.834730    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8bp9\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-kube-api-access-s8bp9\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.834830 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.834758    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbcsm\" (UniqueName: \"kubernetes.io/projected/56add391-4f0b-4bf8-a857-563b82d92fbf-kube-api-access-sbcsm\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:19.834830 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.834798    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a15ef051-198a-49eb-b21d-1d8a7b82ccba-trusted-ca\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.835066 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.834845    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-bound-sa-token\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.835066 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.834872    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/56add391-4f0b-4bf8-a857-563b82d92fbf-config-volume\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:19.835066 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.834905    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a15ef051-198a-49eb-b21d-1d8a7b82ccba-image-registry-private-configuration\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.835066 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.834949    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bff935c-a6ff-41ec-8b79-8146ede38416-cert\") pod \"ingress-canary-5xnh4\" (UID: \"8bff935c-a6ff-41ec-8b79-8146ede38416\") " pod="openshift-ingress-canary/ingress-canary-5xnh4"
Jul 28 17:17:19.835066 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.834972    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:19.835066 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.835038    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56add391-4f0b-4bf8-a857-563b82d92fbf-tmp-dir\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:19.835341 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.835075    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.835341 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.835098    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a15ef051-198a-49eb-b21d-1d8a7b82ccba-ca-trust-extracted\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.835341 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:19.835207    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:19.835341 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.835213    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-certificates\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.835341 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:19.835223    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dfbf9859d-q5c99: secret "image-registry-tls" not found
Jul 28 17:17:19.835341 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:19.835297    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls podName:a15ef051-198a-49eb-b21d-1d8a7b82ccba nodeName:}" failed. No retries permitted until 2026-07-28 17:17:20.335278309 +0000 UTC m=+36.592887983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls") pod "image-registry-dfbf9859d-q5c99" (UID: "a15ef051-198a-49eb-b21d-1d8a7b82ccba") : secret "image-registry-tls" not found
Jul 28 17:17:19.835608 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.835592    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a15ef051-198a-49eb-b21d-1d8a7b82ccba-trusted-ca\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.838888 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.838866    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a15ef051-198a-49eb-b21d-1d8a7b82ccba-image-registry-private-configuration\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.838888 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.838874    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a15ef051-198a-49eb-b21d-1d8a7b82ccba-installation-pull-secrets\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.845196 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.843735    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-bound-sa-token\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.845196 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.844181    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8bp9\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-kube-api-access-s8bp9\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:19.935737 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.935705    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbcsm\" (UniqueName: \"kubernetes.io/projected/56add391-4f0b-4bf8-a857-563b82d92fbf-kube-api-access-sbcsm\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:19.935859 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.935759    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/56add391-4f0b-4bf8-a857-563b82d92fbf-config-volume\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:19.935859 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.935850    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bff935c-a6ff-41ec-8b79-8146ede38416-cert\") pod \"ingress-canary-5xnh4\" (UID: \"8bff935c-a6ff-41ec-8b79-8146ede38416\") " pod="openshift-ingress-canary/ingress-canary-5xnh4"
Jul 28 17:17:19.935945 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.935867    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:19.935945 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.935884    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56add391-4f0b-4bf8-a857-563b82d92fbf-tmp-dir\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:19.935945 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.935919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hmr7\" (UniqueName: \"kubernetes.io/projected/8bff935c-a6ff-41ec-8b79-8146ede38416-kube-api-access-9hmr7\") pod \"ingress-canary-5xnh4\" (UID: \"8bff935c-a6ff-41ec-8b79-8146ede38416\") " pod="openshift-ingress-canary/ingress-canary-5xnh4"
Jul 28 17:17:19.936090 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:19.936020    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:19.936135 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:19.936103    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls podName:56add391-4f0b-4bf8-a857-563b82d92fbf nodeName:}" failed. No retries permitted until 2026-07-28 17:17:20.436081255 +0000 UTC m=+36.693690940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls") pod "dns-default-2nw7c" (UID: "56add391-4f0b-4bf8-a857-563b82d92fbf") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:19.936242 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.936219    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56add391-4f0b-4bf8-a857-563b82d92fbf-tmp-dir\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:19.936328 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.936310    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/56add391-4f0b-4bf8-a857-563b82d92fbf-config-volume\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:19.938119 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.938099    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bff935c-a6ff-41ec-8b79-8146ede38416-cert\") pod \"ingress-canary-5xnh4\" (UID: \"8bff935c-a6ff-41ec-8b79-8146ede38416\") " pod="openshift-ingress-canary/ingress-canary-5xnh4"
Jul 28 17:17:19.946217 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.946194    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbcsm\" (UniqueName: \"kubernetes.io/projected/56add391-4f0b-4bf8-a857-563b82d92fbf-kube-api-access-sbcsm\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:19.946296 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.946192    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hmr7\" (UniqueName: \"kubernetes.io/projected/8bff935c-a6ff-41ec-8b79-8146ede38416-kube-api-access-9hmr7\") pod \"ingress-canary-5xnh4\" (UID: \"8bff935c-a6ff-41ec-8b79-8146ede38416\") " pod="openshift-ingress-canary/ingress-canary-5xnh4"
Jul 28 17:17:19.977268 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:19.977250    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5xnh4"
Jul 28 17:17:20.104422 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:20.104356    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5xnh4"]
Jul 28 17:17:20.107495 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:17:20.107471    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bff935c_a6ff_41ec_8b79_8146ede38416.slice/crio-5e2a6a476c109b4078d7531ea23f6952e1c0877e71cc39d9da9cacccf03ee83f WatchSource:0}: Error finding container 5e2a6a476c109b4078d7531ea23f6952e1c0877e71cc39d9da9cacccf03ee83f: Status 404 returned error can't find the container with id 5e2a6a476c109b4078d7531ea23f6952e1c0877e71cc39d9da9cacccf03ee83f
Jul 28 17:17:20.306931 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:20.306887    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:20.306931 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:20.306933    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:17:20.309727 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:20.309696    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:17:20.309860 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:20.309734    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:17:20.309860 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:20.309839    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2k2q\""
Jul 28 17:17:20.338203 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:20.338173    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:20.338309 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:20.338298    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:20.338309 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:20.338308    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dfbf9859d-q5c99: secret "image-registry-tls" not found
Jul 28 17:17:20.338384 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:20.338362    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls podName:a15ef051-198a-49eb-b21d-1d8a7b82ccba nodeName:}" failed. No retries permitted until 2026-07-28 17:17:21.338348625 +0000 UTC m=+37.595958293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls") pod "image-registry-dfbf9859d-q5c99" (UID: "a15ef051-198a-49eb-b21d-1d8a7b82ccba") : secret "image-registry-tls" not found
Jul 28 17:17:20.438945 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:20.438912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:20.439093 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:20.439005    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:20.439093 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:20.439055    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls podName:56add391-4f0b-4bf8-a857-563b82d92fbf nodeName:}" failed. No retries permitted until 2026-07-28 17:17:21.439040197 +0000 UTC m=+37.696649869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls") pod "dns-default-2nw7c" (UID: "56add391-4f0b-4bf8-a857-563b82d92fbf") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:20.549755 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:20.549727    2575 generic.go:356] "Generic (PLEG): container finished" podID="325554ac-10f4-4076-b562-20b27ddf18b0" containerID="4412daa054ebd380e3e2311fa6ee5d1b403206afb5d91002e7df103b65fc03bf" exitCode=0
Jul 28 17:17:20.549869 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:20.549819    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnpnk" event={"ID":"325554ac-10f4-4076-b562-20b27ddf18b0","Type":"ContainerDied","Data":"4412daa054ebd380e3e2311fa6ee5d1b403206afb5d91002e7df103b65fc03bf"}
Jul 28 17:17:20.550989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:20.550970    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5xnh4" event={"ID":"8bff935c-a6ff-41ec-8b79-8146ede38416","Type":"ContainerStarted","Data":"5e2a6a476c109b4078d7531ea23f6952e1c0877e71cc39d9da9cacccf03ee83f"}
Jul 28 17:17:21.306437 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:21.306189    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:21.309314 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:21.309287    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:17:21.310177 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:21.309916    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:17:21.310177 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:21.309943    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wqrl8\""
Jul 28 17:17:21.345127 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:21.345092    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:21.345607 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:21.345271    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:21.345607 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:21.345293    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dfbf9859d-q5c99: secret "image-registry-tls" not found
Jul 28 17:17:21.345607 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:21.345362    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls podName:a15ef051-198a-49eb-b21d-1d8a7b82ccba nodeName:}" failed. No retries permitted until 2026-07-28 17:17:23.345334796 +0000 UTC m=+39.602944485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls") pod "image-registry-dfbf9859d-q5c99" (UID: "a15ef051-198a-49eb-b21d-1d8a7b82ccba") : secret "image-registry-tls" not found
Jul 28 17:17:21.445793 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:21.445762    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:21.445947 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:21.445926    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:21.446002 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:21.445995    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls podName:56add391-4f0b-4bf8-a857-563b82d92fbf nodeName:}" failed. No retries permitted until 2026-07-28 17:17:23.445979001 +0000 UTC m=+39.703588669 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls") pod "dns-default-2nw7c" (UID: "56add391-4f0b-4bf8-a857-563b82d92fbf") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:21.557592 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:21.557471    2575 generic.go:356] "Generic (PLEG): container finished" podID="325554ac-10f4-4076-b562-20b27ddf18b0" containerID="0e6f9331073cd56990b3e70c01a4ad8e022135eb8d25eb716cfac9a7f131f0b4" exitCode=0
Jul 28 17:17:21.557592 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:21.557531    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnpnk" event={"ID":"325554ac-10f4-4076-b562-20b27ddf18b0","Type":"ContainerDied","Data":"0e6f9331073cd56990b3e70c01a4ad8e022135eb8d25eb716cfac9a7f131f0b4"}
Jul 28 17:17:22.564232 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:22.564198    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnpnk" event={"ID":"325554ac-10f4-4076-b562-20b27ddf18b0","Type":"ContainerStarted","Data":"9c21fa3c720bf4323497ab5b08ee1183dc9ccb253b1e4a8e0b5655157848a163"}
Jul 28 17:17:22.597052 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:22.596990    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xnpnk" podStartSLOduration=6.577732307 podStartE2EDuration="37.59697116s" totalImagesPullingTime="31.019238853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:22.595057899 +0000 UTC m=+38.852667590" watchObservedRunningTime="2026-07-28 17:17:22.59697116 +0000 UTC m=+38.854580849"
Jul 28 17:17:23.359381 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:23.359297    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:23.359540 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:23.359448    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:23.359540 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:23.359463    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dfbf9859d-q5c99: secret "image-registry-tls" not found
Jul 28 17:17:23.359540 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:23.359517    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls podName:a15ef051-198a-49eb-b21d-1d8a7b82ccba nodeName:}" failed. No retries permitted until 2026-07-28 17:17:27.359502148 +0000 UTC m=+43.617111820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls") pod "image-registry-dfbf9859d-q5c99" (UID: "a15ef051-198a-49eb-b21d-1d8a7b82ccba") : secret "image-registry-tls" not found
Jul 28 17:17:23.459924 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:23.459893    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:23.460065 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:23.460045    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:23.460124 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:23.460115    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls podName:56add391-4f0b-4bf8-a857-563b82d92fbf nodeName:}" failed. No retries permitted until 2026-07-28 17:17:27.460095751 +0000 UTC m=+43.717705425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls") pod "dns-default-2nw7c" (UID: "56add391-4f0b-4bf8-a857-563b82d92fbf") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:23.567667 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:23.567630    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5xnh4" event={"ID":"8bff935c-a6ff-41ec-8b79-8146ede38416","Type":"ContainerStarted","Data":"05f18cd96d975b09027c49e60a6df7a725f0cd59a332543ebdfd6a4fd8c4a7f2"}
Jul 28 17:17:23.599510 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:23.599462    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5xnh4" podStartSLOduration=1.676225911 podStartE2EDuration="4.599445613s" totalImagesPullingTime="2.923219702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:23.599302668 +0000 UTC m=+39.856912359" watchObservedRunningTime="2026-07-28 17:17:23.599445613 +0000 UTC m=+39.857055302"
Jul 28 17:17:27.386643 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:27.386602    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:27.387026 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:27.386777    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:27.387026 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:27.386796    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dfbf9859d-q5c99: secret "image-registry-tls" not found
Jul 28 17:17:27.387026 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:27.386853    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls podName:a15ef051-198a-49eb-b21d-1d8a7b82ccba nodeName:}" failed. No retries permitted until 2026-07-28 17:17:35.386837761 +0000 UTC m=+51.644447429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls") pod "image-registry-dfbf9859d-q5c99" (UID: "a15ef051-198a-49eb-b21d-1d8a7b82ccba") : secret "image-registry-tls" not found
Jul 28 17:17:27.487652 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:27.487612    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:27.487810 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:27.487749    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:27.487863 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:27.487814    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls podName:56add391-4f0b-4bf8-a857-563b82d92fbf nodeName:}" failed. No retries permitted until 2026-07-28 17:17:35.487796623 +0000 UTC m=+51.745406298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls") pod "dns-default-2nw7c" (UID: "56add391-4f0b-4bf8-a857-563b82d92fbf") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:35.444664 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:35.444621    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:35.445174 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:35.444771    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:35.445174 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:35.444788    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dfbf9859d-q5c99: secret "image-registry-tls" not found
Jul 28 17:17:35.445174 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:35.444843    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls podName:a15ef051-198a-49eb-b21d-1d8a7b82ccba nodeName:}" failed. No retries permitted until 2026-07-28 17:17:51.444827037 +0000 UTC m=+67.702436710 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls") pod "image-registry-dfbf9859d-q5c99" (UID: "a15ef051-198a-49eb-b21d-1d8a7b82ccba") : secret "image-registry-tls" not found
Jul 28 17:17:35.545573 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:35.545535    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:35.545755 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:35.545688    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:35.545822 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:35.545763    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls podName:56add391-4f0b-4bf8-a857-563b82d92fbf nodeName:}" failed. No retries permitted until 2026-07-28 17:17:51.545744183 +0000 UTC m=+67.803353851 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls") pod "dns-default-2nw7c" (UID: "56add391-4f0b-4bf8-a857-563b82d92fbf") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:43.839344 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.839308    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-mh7dr"]
Jul 28 17:17:43.841681 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.841666    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mh7dr"
Jul 28 17:17:43.844597 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.843969    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"]
Jul 28 17:17:43.844597 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.844167    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-45j56\""
Jul 28 17:17:43.846132 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.846112    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-vkqmg"]
Jul 28 17:17:43.846257 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.846240    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:17:43.848303 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.848281    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-c679z"]
Jul 28 17:17:43.848419 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.848403    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:43.849453 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.849428    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:17:43.849581 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.849571    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 28 17:17:43.849724 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.849706    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8lxpl\""
Jul 28 17:17:43.850178 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.849890    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 28 17:17:43.850178 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.850003    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:17:43.850303 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.850207    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:43.850638 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.850615    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 28 17:17:43.850638 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.850634    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 28 17:17:43.850788 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.850758    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:17:43.850842 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.850805    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nnmsx\""
Jul 28 17:17:43.850894 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.850870    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:17:43.853190 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.853161    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:17:43.853190 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.853182    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 28 17:17:43.853340 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.853197    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-blwhd\""
Jul 28 17:17:43.853662 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.853643    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-mh7dr"]
Jul 28 17:17:43.854017 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.853997    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 28 17:17:43.854130 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.854031    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 28 17:17:43.858436 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.858412    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"]
Jul 28 17:17:43.859351 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.859324    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 28 17:17:43.860311 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.860291    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 28 17:17:43.862593 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.862562    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-vkqmg"]
Jul 28 17:17:43.863324 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.863303    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-c679z"]
Jul 28 17:17:43.905728 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.905707    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7-config\") pod \"console-operator-584dcb6b54-c679z\" (UID: \"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7\") " pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:43.905853 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.905735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97z97\" (UniqueName: \"kubernetes.io/projected/958ca081-580a-4d51-af0a-093e902e405e-kube-api-access-97z97\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:43.905853 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.905753    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7-serving-cert\") pod \"console-operator-584dcb6b54-c679z\" (UID: \"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7\") " pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:43.905853 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.905776    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52mm2\" (UniqueName: \"kubernetes.io/projected/04d98107-270d-46aa-b9e4-013adb971e13-kube-api-access-52mm2\") pod \"cluster-monitoring-operator-548b489cb8-gd67w\" (UID: \"04d98107-270d-46aa-b9e4-013adb971e13\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:17:43.905853 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.905821    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/04d98107-270d-46aa-b9e4-013adb971e13-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-gd67w\" (UID: \"04d98107-270d-46aa-b9e4-013adb971e13\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:17:43.906031 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.905868    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9strw\" (UniqueName: \"kubernetes.io/projected/1f11e547-c105-4e8d-a652-0d862416bcd6-kube-api-access-9strw\") pod \"network-check-source-7d88cb55d8-mh7dr\" (UID: \"1f11e547-c105-4e8d-a652-0d862416bcd6\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mh7dr"
Jul 28 17:17:43.906031 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.905887    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/958ca081-580a-4d51-af0a-093e902e405e-service-ca-bundle\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:43.906031 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.905912    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvmr\" (UniqueName: \"kubernetes.io/projected/7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7-kube-api-access-stvmr\") pod \"console-operator-584dcb6b54-c679z\" (UID: \"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7\") " pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:43.906031 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.905939    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-gd67w\" (UID: \"04d98107-270d-46aa-b9e4-013adb971e13\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:17:43.906031 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.905956    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/958ca081-580a-4d51-af0a-093e902e405e-tmp\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:43.906031 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.905993    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7-trusted-ca\") pod \"console-operator-584dcb6b54-c679z\" (UID: \"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7\") " pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:43.906031 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.906008    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/958ca081-580a-4d51-af0a-093e902e405e-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:43.906031 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.906024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/958ca081-580a-4d51-af0a-093e902e405e-serving-cert\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:43.906270 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.906079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/958ca081-580a-4d51-af0a-093e902e405e-snapshots\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:43.957286 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.957258    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-fsv8l"]
Jul 28 17:17:43.959273 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.959258    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-fsv8l"
Jul 28 17:17:43.961673 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.961652    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zxxwm\""
Jul 28 17:17:43.961770 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.961686    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 17:17:43.961770 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.961686    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:17:43.972703 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:43.972685    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-fsv8l"]
Jul 28 17:17:44.007007 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.006981    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9strw\" (UniqueName: \"kubernetes.io/projected/1f11e547-c105-4e8d-a652-0d862416bcd6-kube-api-access-9strw\") pod \"network-check-source-7d88cb55d8-mh7dr\" (UID: \"1f11e547-c105-4e8d-a652-0d862416bcd6\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mh7dr"
Jul 28 17:17:44.007107 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007013    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/958ca081-580a-4d51-af0a-093e902e405e-service-ca-bundle\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:44.007107 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007033    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvmr\" (UniqueName: \"kubernetes.io/projected/7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7-kube-api-access-stvmr\") pod \"console-operator-584dcb6b54-c679z\" (UID: \"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7\") " pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:44.007107 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007057    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-gd67w\" (UID: \"04d98107-270d-46aa-b9e4-013adb971e13\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:17:44.007254 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007100    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/958ca081-580a-4d51-af0a-093e902e405e-tmp\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:44.007254 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:44.007144    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:17:44.007254 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7-trusted-ca\") pod \"console-operator-584dcb6b54-c679z\" (UID: \"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7\") " pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:44.007254 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007181    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/958ca081-580a-4d51-af0a-093e902e405e-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:44.007254 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:44.007199    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls podName:04d98107-270d-46aa-b9e4-013adb971e13 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:44.507180255 +0000 UTC m=+60.764789943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-gd67w" (UID: "04d98107-270d-46aa-b9e4-013adb971e13") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:17:44.007254 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007225    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/958ca081-580a-4d51-af0a-093e902e405e-serving-cert\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:44.007597 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007279    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/958ca081-580a-4d51-af0a-093e902e405e-snapshots\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:44.007597 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007328    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7-config\") pod \"console-operator-584dcb6b54-c679z\" (UID: \"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7\") " pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:44.007597 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97z97\" (UniqueName: \"kubernetes.io/projected/958ca081-580a-4d51-af0a-093e902e405e-kube-api-access-97z97\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:44.007597 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7-serving-cert\") pod \"console-operator-584dcb6b54-c679z\" (UID: \"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7\") " pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:44.007875 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007831    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52mm2\" (UniqueName: \"kubernetes.io/projected/04d98107-270d-46aa-b9e4-013adb971e13-kube-api-access-52mm2\") pod \"cluster-monitoring-operator-548b489cb8-gd67w\" (UID: \"04d98107-270d-46aa-b9e4-013adb971e13\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:17:44.007981 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhg2w\" (UniqueName: \"kubernetes.io/projected/e9a31f62-4a8d-48c8-8e8a-73079127d4d1-kube-api-access-zhg2w\") pod \"volume-data-source-validator-549c6d54cf-fsv8l\" (UID: \"e9a31f62-4a8d-48c8-8e8a-73079127d4d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-fsv8l"
Jul 28 17:17:44.007981 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007936    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/04d98107-270d-46aa-b9e4-013adb971e13-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-gd67w\" (UID: \"04d98107-270d-46aa-b9e4-013adb971e13\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:17:44.007981 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007946    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/958ca081-580a-4d51-af0a-093e902e405e-service-ca-bundle\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:44.008125 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.007988    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/958ca081-580a-4d51-af0a-093e902e405e-tmp\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:44.008125 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.008026    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7-config\") pod \"console-operator-584dcb6b54-c679z\" (UID: \"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7\") " pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:44.008266 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.008209    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/958ca081-580a-4d51-af0a-093e902e405e-snapshots\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:44.008429 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.008274    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7-trusted-ca\") pod \"console-operator-584dcb6b54-c679z\" (UID: \"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7\") " pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:44.008625 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.008605    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/958ca081-580a-4d51-af0a-093e902e405e-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:44.008709 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.008639    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/04d98107-270d-46aa-b9e4-013adb971e13-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-gd67w\" (UID: \"04d98107-270d-46aa-b9e4-013adb971e13\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:17:44.010643 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.010628    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/958ca081-580a-4d51-af0a-093e902e405e-serving-cert\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:44.010643 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.010633    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7-serving-cert\") pod \"console-operator-584dcb6b54-c679z\" (UID: \"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7\") " pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:44.017584 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.017561    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97z97\" (UniqueName: \"kubernetes.io/projected/958ca081-580a-4d51-af0a-093e902e405e-kube-api-access-97z97\") pod \"insights-operator-5885fdb759-vkqmg\" (UID: \"958ca081-580a-4d51-af0a-093e902e405e\") " pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:44.018692 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.018662    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9strw\" (UniqueName: \"kubernetes.io/projected/1f11e547-c105-4e8d-a652-0d862416bcd6-kube-api-access-9strw\") pod \"network-check-source-7d88cb55d8-mh7dr\" (UID: \"1f11e547-c105-4e8d-a652-0d862416bcd6\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mh7dr"
Jul 28 17:17:44.018776 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.018760    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvmr\" (UniqueName: \"kubernetes.io/projected/7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7-kube-api-access-stvmr\") pod \"console-operator-584dcb6b54-c679z\" (UID: \"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7\") " pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:44.018895 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.018874    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52mm2\" (UniqueName: \"kubernetes.io/projected/04d98107-270d-46aa-b9e4-013adb971e13-kube-api-access-52mm2\") pod \"cluster-monitoring-operator-548b489cb8-gd67w\" (UID: \"04d98107-270d-46aa-b9e4-013adb971e13\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:17:44.109274 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.109218    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhg2w\" (UniqueName: \"kubernetes.io/projected/e9a31f62-4a8d-48c8-8e8a-73079127d4d1-kube-api-access-zhg2w\") pod \"volume-data-source-validator-549c6d54cf-fsv8l\" (UID: \"e9a31f62-4a8d-48c8-8e8a-73079127d4d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-fsv8l"
Jul 28 17:17:44.131431 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.131410    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 17:17:44.145451 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.145431    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:17:44.149693 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.149672    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhg2w\" (UniqueName: \"kubernetes.io/projected/e9a31f62-4a8d-48c8-8e8a-73079127d4d1-kube-api-access-zhg2w\") pod \"volume-data-source-validator-549c6d54cf-fsv8l\" (UID: \"e9a31f62-4a8d-48c8-8e8a-73079127d4d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-fsv8l"
Jul 28 17:17:44.158730 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.158710    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-45j56\""
Jul 28 17:17:44.163766 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.163749    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mh7dr"
Jul 28 17:17:44.170658 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.170637    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nnmsx\""
Jul 28 17:17:44.178089 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.178072    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-vkqmg"
Jul 28 17:17:44.178411 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.178395    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-blwhd\""
Jul 28 17:17:44.187242 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.187221    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:44.270442 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.270375    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zxxwm\""
Jul 28 17:17:44.278000 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.277886    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-fsv8l"
Jul 28 17:17:44.320340 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.320225    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-mh7dr"]
Jul 28 17:17:44.322608 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:17:44.322578    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f11e547_c105_4e8d_a652_0d862416bcd6.slice/crio-0823d3a6835e5e88ffa84c3707896d10041681bbaba09df1e6e5c08e6caa437e WatchSource:0}: Error finding container 0823d3a6835e5e88ffa84c3707896d10041681bbaba09df1e6e5c08e6caa437e: Status 404 returned error can't find the container with id 0823d3a6835e5e88ffa84c3707896d10041681bbaba09df1e6e5c08e6caa437e
Jul 28 17:17:44.337879 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.337851    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-vkqmg"]
Jul 28 17:17:44.340815 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:17:44.340788    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958ca081_580a_4d51_af0a_093e902e405e.slice/crio-409cbdb8990a892876a1575f2a8decb8d3c9097f36aa1fb03c4509a3d3762f7b WatchSource:0}: Error finding container 409cbdb8990a892876a1575f2a8decb8d3c9097f36aa1fb03c4509a3d3762f7b: Status 404 returned error can't find the container with id 409cbdb8990a892876a1575f2a8decb8d3c9097f36aa1fb03c4509a3d3762f7b
Jul 28 17:17:44.348029 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.348003    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-c679z"]
Jul 28 17:17:44.351693 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:17:44.351666    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1c7280_90b9_49e0_bd9b_6c91ca7e1ff7.slice/crio-c90280f7d3bb46264797738099851a280ec3ed2a5b4278bed98c358a46b4b5cf WatchSource:0}: Error finding container c90280f7d3bb46264797738099851a280ec3ed2a5b4278bed98c358a46b4b5cf: Status 404 returned error can't find the container with id c90280f7d3bb46264797738099851a280ec3ed2a5b4278bed98c358a46b4b5cf
Jul 28 17:17:44.413239 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.413211    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-fsv8l"]
Jul 28 17:17:44.416185 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:17:44.416160    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a31f62_4a8d_48c8_8e8a_73079127d4d1.slice/crio-dd6f945810f038527c25a4818efa244fe29b1e2e55e808656e1167e6fdd4a107 WatchSource:0}: Error finding container dd6f945810f038527c25a4818efa244fe29b1e2e55e808656e1167e6fdd4a107: Status 404 returned error can't find the container with id dd6f945810f038527c25a4818efa244fe29b1e2e55e808656e1167e6fdd4a107
Jul 28 17:17:44.513906 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.513877    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-gd67w\" (UID: \"04d98107-270d-46aa-b9e4-013adb971e13\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:17:44.514083 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:44.514053    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:17:44.514151 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:44.514128    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls podName:04d98107-270d-46aa-b9e4-013adb971e13 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:45.514108344 +0000 UTC m=+61.771718030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-gd67w" (UID: "04d98107-270d-46aa-b9e4-013adb971e13") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:17:44.547107 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.547086    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c954x"
Jul 28 17:17:44.613992 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.613954    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mh7dr" event={"ID":"1f11e547-c105-4e8d-a652-0d862416bcd6","Type":"ContainerStarted","Data":"0823d3a6835e5e88ffa84c3707896d10041681bbaba09df1e6e5c08e6caa437e"}
Jul 28 17:17:44.615128 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.615105    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-c679z" event={"ID":"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7","Type":"ContainerStarted","Data":"c90280f7d3bb46264797738099851a280ec3ed2a5b4278bed98c358a46b4b5cf"}
Jul 28 17:17:44.616101 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.616079    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-vkqmg" event={"ID":"958ca081-580a-4d51-af0a-093e902e405e","Type":"ContainerStarted","Data":"409cbdb8990a892876a1575f2a8decb8d3c9097f36aa1fb03c4509a3d3762f7b"}
Jul 28 17:17:44.617118 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:44.617099    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-fsv8l" event={"ID":"e9a31f62-4a8d-48c8-8e8a-73079127d4d1","Type":"ContainerStarted","Data":"dd6f945810f038527c25a4818efa244fe29b1e2e55e808656e1167e6fdd4a107"}
Jul 28 17:17:45.521428 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:45.520863    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-gd67w\" (UID: \"04d98107-270d-46aa-b9e4-013adb971e13\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:17:45.521428 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:45.521008    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:17:45.521428 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:45.521080    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls podName:04d98107-270d-46aa-b9e4-013adb971e13 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:47.521059942 +0000 UTC m=+63.778669657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-gd67w" (UID: "04d98107-270d-46aa-b9e4-013adb971e13") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:17:47.537665 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:47.537624    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-gd67w\" (UID: \"04d98107-270d-46aa-b9e4-013adb971e13\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:17:47.538254 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:47.537792    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:17:47.538254 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:47.537853    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls podName:04d98107-270d-46aa-b9e4-013adb971e13 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:51.537838422 +0000 UTC m=+67.795448092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-gd67w" (UID: "04d98107-270d-46aa-b9e4-013adb971e13") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:17:48.629641 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:48.629600    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mh7dr" event={"ID":"1f11e547-c105-4e8d-a652-0d862416bcd6","Type":"ContainerStarted","Data":"850f9da83dd62ee1c198975cda85aea98d4b2fea961a142f54748c9a91a75c51"}
Jul 28 17:17:48.631184 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:48.631161    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/0.log"
Jul 28 17:17:48.631305 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:48.631201    2575 generic.go:356] "Generic (PLEG): container finished" podID="7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7" containerID="74c4e5a116cdb7f5a2586edbc495eca25f5bdfd0513a0fc9637358f8deaff3de" exitCode=255
Jul 28 17:17:48.631305 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:48.631283    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-c679z" event={"ID":"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7","Type":"ContainerDied","Data":"74c4e5a116cdb7f5a2586edbc495eca25f5bdfd0513a0fc9637358f8deaff3de"}
Jul 28 17:17:48.631472 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:48.631453    2575 scope.go:117] "RemoveContainer" containerID="74c4e5a116cdb7f5a2586edbc495eca25f5bdfd0513a0fc9637358f8deaff3de"
Jul 28 17:17:48.637874 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:48.637853    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-vkqmg" event={"ID":"958ca081-580a-4d51-af0a-093e902e405e","Type":"ContainerStarted","Data":"75054e9d01991de34e20451b41f5d6affcc575009df4b92bc6dab1c61b033041"}
Jul 28 17:17:48.639354 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:48.639335    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-fsv8l" event={"ID":"e9a31f62-4a8d-48c8-8e8a-73079127d4d1","Type":"ContainerStarted","Data":"ad3c471b09f1d3b2a38bef0f7fb468da0e9b1d28f8c520cd39ed5838594b8078"}
Jul 28 17:17:48.776116 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:48.776067    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mh7dr" podStartSLOduration=2.223740802 podStartE2EDuration="5.776052227s" totalImagesPullingTime="3.552311425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:48.732780753 +0000 UTC m=+64.990390443" watchObservedRunningTime="2026-07-28 17:17:48.776052227 +0000 UTC m=+65.033661938"
Jul 28 17:17:48.776499 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:48.776460    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-vkqmg" podStartSLOduration=2.256341644 podStartE2EDuration="5.77644842s" totalImagesPullingTime="3.520106776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:48.775915719 +0000 UTC m=+65.033525410" watchObservedRunningTime="2026-07-28 17:17:48.77644842 +0000 UTC m=+65.034058111"
Jul 28 17:17:48.795256 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:48.795214    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-fsv8l" podStartSLOduration=2.35179946 podStartE2EDuration="5.795199223s" totalImagesPullingTime="3.443399763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:48.794506216 +0000 UTC m=+65.052115931" watchObservedRunningTime="2026-07-28 17:17:48.795199223 +0000 UTC m=+65.052808914"
Jul 28 17:17:48.948210 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:48.948173    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5shf\" (UniqueName: \"kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf\") pod \"network-check-target-n6cbg\" (UID: \"e041de26-66e3-4855-9270-cf54fe1d8045\") " pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:48.950571 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:48.950549    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5shf\" (UniqueName: \"kubernetes.io/projected/e041de26-66e3-4855-9270-cf54fe1d8045-kube-api-access-l5shf\") pod \"network-check-target-n6cbg\" (UID: \"e041de26-66e3-4855-9270-cf54fe1d8045\") " pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:49.049025 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.048981    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs\") pod \"network-metrics-daemon-zqjzm\" (UID: \"0ee703af-bd7c-45e1-8b8a-1b31645d40bf\") " pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:17:49.049212 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.049089    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:49.051339 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.051313    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:17:49.051464 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.051337    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:17:49.059252 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:49.059219    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 17:17:49.059368 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:49.059274    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs podName:0ee703af-bd7c-45e1-8b8a-1b31645d40bf nodeName:}" failed. No retries permitted until 2026-07-28 17:18:53.059259269 +0000 UTC m=+129.316868938 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs") pod "network-metrics-daemon-zqjzm" (UID: "0ee703af-bd7c-45e1-8b8a-1b31645d40bf") : secret "metrics-daemon-secret" not found
Jul 28 17:17:49.061446 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.061419    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de9f57b-63bf-4dad-be1a-86716a7758f8-original-pull-secret\") pod \"global-pull-secret-syncer-8sv5v\" (UID: \"2de9f57b-63bf-4dad-be1a-86716a7758f8\") " pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:49.117405 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.117369    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv5v"
Jul 28 17:17:49.221085 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.221007    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wqrl8\""
Jul 28 17:17:49.229027 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.229002    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:49.240033 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.240007    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8sv5v"]
Jul 28 17:17:49.243654 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:17:49.243630    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de9f57b_63bf_4dad_be1a_86716a7758f8.slice/crio-c91a4ee10074c69239534dbca97a2a6c197f0d6396750e7910df6efe61d17485 WatchSource:0}: Error finding container c91a4ee10074c69239534dbca97a2a6c197f0d6396750e7910df6efe61d17485: Status 404 returned error can't find the container with id c91a4ee10074c69239534dbca97a2a6c197f0d6396750e7910df6efe61d17485
Jul 28 17:17:49.310513 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.310469    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zp6vz"]
Jul 28 17:17:49.313538 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.313503    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.315786 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.315765    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:17:49.316196 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.316071    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:17:49.316295 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.316191    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hbmbv\""
Jul 28 17:17:49.326186 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.326110    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zp6vz"]
Jul 28 17:17:49.352311 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.352281    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj2vl\" (UniqueName: \"kubernetes.io/projected/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-api-access-nj2vl\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.352459 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.352343    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-crio-socket\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.352459 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.352428    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.352602 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.352466    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.352602 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.352487    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-data-volume\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.378986 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.378955    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n6cbg"]
Jul 28 17:17:49.382277 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:17:49.382252    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode041de26_66e3_4855_9270_cf54fe1d8045.slice/crio-7e627382defc640b9313a1b18c366e495b7766ae44f8795d165aa70841260a71 WatchSource:0}: Error finding container 7e627382defc640b9313a1b18c366e495b7766ae44f8795d165aa70841260a71: Status 404 returned error can't find the container with id 7e627382defc640b9313a1b18c366e495b7766ae44f8795d165aa70841260a71
Jul 28 17:17:49.453598 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.453574    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-crio-socket\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.453709 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.453642    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.453709 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.453687    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-crio-socket\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.453796 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.453691    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.453796 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.453732    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-data-volume\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.453796 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.453777    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj2vl\" (UniqueName: \"kubernetes.io/projected/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-api-access-nj2vl\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.453902 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:49.453780    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:49.453902 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:49.453884    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls podName:269dc00a-7bd4-4a18-97a3-2a1ec916dd93 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:49.95386588 +0000 UTC m=+66.211475560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zp6vz" (UID: "269dc00a-7bd4-4a18-97a3-2a1ec916dd93") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:49.453902 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:49.453805    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:49.454008 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:49.453939    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm podName:269dc00a-7bd4-4a18-97a3-2a1ec916dd93 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:49.953928419 +0000 UTC m=+66.211538108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zp6vz" (UID: "269dc00a-7bd4-4a18-97a3-2a1ec916dd93") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:49.454054 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.454025    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-data-volume\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.461957 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.461938    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj2vl\" (UniqueName: \"kubernetes.io/projected/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-api-access-nj2vl\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.645022 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.644918    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8sv5v" event={"ID":"2de9f57b-63bf-4dad-be1a-86716a7758f8","Type":"ContainerStarted","Data":"c91a4ee10074c69239534dbca97a2a6c197f0d6396750e7910df6efe61d17485"}
Jul 28 17:17:49.646882 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.646850    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n6cbg" event={"ID":"e041de26-66e3-4855-9270-cf54fe1d8045","Type":"ContainerStarted","Data":"eb7067f2aa6752a42f19f51a6b3531fe213bee47e6f2f93c9cd5e19fdef48fe2"}
Jul 28 17:17:49.647021 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.646894    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n6cbg" event={"ID":"e041de26-66e3-4855-9270-cf54fe1d8045","Type":"ContainerStarted","Data":"7e627382defc640b9313a1b18c366e495b7766ae44f8795d165aa70841260a71"}
Jul 28 17:17:49.647021 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.646953    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:17:49.648793 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.648771    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:17:49.649287 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.649241    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/0.log"
Jul 28 17:17:49.649374 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.649293    2575 generic.go:356] "Generic (PLEG): container finished" podID="7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7" containerID="7ff450761ab748ca76ba7abb6c877ce163a2a5de7ca05e7ee3995b611f6165b1" exitCode=255
Jul 28 17:17:49.649433 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.649411    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-c679z" event={"ID":"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7","Type":"ContainerDied","Data":"7ff450761ab748ca76ba7abb6c877ce163a2a5de7ca05e7ee3995b611f6165b1"}
Jul 28 17:17:49.649486 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.649467    2575 scope.go:117] "RemoveContainer" containerID="74c4e5a116cdb7f5a2586edbc495eca25f5bdfd0513a0fc9637358f8deaff3de"
Jul 28 17:17:49.649763 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.649742    2575 scope.go:117] "RemoveContainer" containerID="7ff450761ab748ca76ba7abb6c877ce163a2a5de7ca05e7ee3995b611f6165b1"
Jul 28 17:17:49.649937 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:49.649912    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-c679z_openshift-console-operator(7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7)\"" pod="openshift-console-operator/console-operator-584dcb6b54-c679z" podUID="7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7"
Jul 28 17:17:49.664001 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.663961    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n6cbg" podStartSLOduration=65.663942703 podStartE2EDuration="1m5.663942703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:49.662946229 +0000 UTC m=+65.920555918" watchObservedRunningTime="2026-07-28 17:17:49.663942703 +0000 UTC m=+65.921552395"
Jul 28 17:17:49.958539 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.958487    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.958714 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:49.958572    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:49.958714 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:49.958639    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:49.958714 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:49.958639    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:49.958865 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:49.958709    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls podName:269dc00a-7bd4-4a18-97a3-2a1ec916dd93 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:50.958691658 +0000 UTC m=+67.216301351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zp6vz" (UID: "269dc00a-7bd4-4a18-97a3-2a1ec916dd93") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:49.958865 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:49.958801    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm podName:269dc00a-7bd4-4a18-97a3-2a1ec916dd93 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:50.958779806 +0000 UTC m=+67.216389474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zp6vz" (UID: "269dc00a-7bd4-4a18-97a3-2a1ec916dd93") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:50.654960 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:50.654926    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:17:50.655345 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:50.655309    2575 scope.go:117] "RemoveContainer" containerID="7ff450761ab748ca76ba7abb6c877ce163a2a5de7ca05e7ee3995b611f6165b1"
Jul 28 17:17:50.655491 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:50.655472    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-c679z_openshift-console-operator(7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7)\"" pod="openshift-console-operator/console-operator-584dcb6b54-c679z" podUID="7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7"
Jul 28 17:17:50.965781 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:50.965750    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:50.965949 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:50.965797    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:50.965949 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:50.965890    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:50.965949 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:50.965915    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:50.965949 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:50.965948    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm podName:269dc00a-7bd4-4a18-97a3-2a1ec916dd93 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:52.965931605 +0000 UTC m=+69.223541298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zp6vz" (UID: "269dc00a-7bd4-4a18-97a3-2a1ec916dd93") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:50.966115 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:50.965968    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls podName:269dc00a-7bd4-4a18-97a3-2a1ec916dd93 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:52.965954847 +0000 UTC m=+69.223564520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zp6vz" (UID: "269dc00a-7bd4-4a18-97a3-2a1ec916dd93") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:51.469265 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:51.469229    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:17:51.469443 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:51.469406    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:51.469443 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:51.469431    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dfbf9859d-q5c99: secret "image-registry-tls" not found
Jul 28 17:17:51.469553 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:51.469502    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls podName:a15ef051-198a-49eb-b21d-1d8a7b82ccba nodeName:}" failed. No retries permitted until 2026-07-28 17:18:23.469482191 +0000 UTC m=+99.727091864 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls") pod "image-registry-dfbf9859d-q5c99" (UID: "a15ef051-198a-49eb-b21d-1d8a7b82ccba") : secret "image-registry-tls" not found
Jul 28 17:17:51.523038 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:51.523010    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gcwqn_bac64312-87a3-4c2c-8cfa-80754d0123ed/dns-node-resolver/0.log"
Jul 28 17:17:51.570030 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:51.569991    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:17:51.570197 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:51.570072    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-gd67w\" (UID: \"04d98107-270d-46aa-b9e4-013adb971e13\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:17:51.570197 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:51.570166    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:17:51.570197 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:51.570173    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:51.570346 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:51.570220    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls podName:04d98107-270d-46aa-b9e4-013adb971e13 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:59.570204251 +0000 UTC m=+75.827813919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-gd67w" (UID: "04d98107-270d-46aa-b9e4-013adb971e13") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:17:51.570346 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:51.570247    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls podName:56add391-4f0b-4bf8-a857-563b82d92fbf nodeName:}" failed. No retries permitted until 2026-07-28 17:18:23.570229122 +0000 UTC m=+99.827838791 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls") pod "dns-default-2nw7c" (UID: "56add391-4f0b-4bf8-a857-563b82d92fbf") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:52.322174 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:52.322144    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wptfc_f4350004-c24b-49f4-9142-b7d268dd1132/node-ca/0.log"
Jul 28 17:17:52.921936 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:52.921906    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5xnh4_8bff935c-a6ff-41ec-8b79-8146ede38416/serve-healthcheck-canary/0.log"
Jul 28 17:17:52.980293 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:52.980254    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:52.980475 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:52.980375    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:52.980475 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:52.980382    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:52.980475 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:52.980445    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm podName:269dc00a-7bd4-4a18-97a3-2a1ec916dd93 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:56.980425899 +0000 UTC m=+73.238035568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zp6vz" (UID: "269dc00a-7bd4-4a18-97a3-2a1ec916dd93") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:52.980707 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:52.980489    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:52.980707 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:52.980568    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls podName:269dc00a-7bd4-4a18-97a3-2a1ec916dd93 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:56.980550474 +0000 UTC m=+73.238160146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zp6vz" (UID: "269dc00a-7bd4-4a18-97a3-2a1ec916dd93") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:53.667159 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:53.667125    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8sv5v" event={"ID":"2de9f57b-63bf-4dad-be1a-86716a7758f8","Type":"ContainerStarted","Data":"874ee628e5cb2c6a7203018b976e140718a6d36c7df27bb456f32bf3b0a3a156"}
Jul 28 17:17:53.682134 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:53.682093    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8sv5v" podStartSLOduration=64.792050317 podStartE2EDuration="1m8.682078514s" totalImagesPullingTime="3.890028197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:53.681426367 +0000 UTC m=+69.939036057" watchObservedRunningTime="2026-07-28 17:17:53.682078514 +0000 UTC m=+69.939688204"
Jul 28 17:17:54.187857 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:54.187829    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:54.188005 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:54.187891    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:17:54.188204 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:54.188191    2575 scope.go:117] "RemoveContainer" containerID="7ff450761ab748ca76ba7abb6c877ce163a2a5de7ca05e7ee3995b611f6165b1"
Jul 28 17:17:54.188355 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:54.188333    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-c679z_openshift-console-operator(7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7)\"" pod="openshift-console-operator/console-operator-584dcb6b54-c679z" podUID="7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7"
Jul 28 17:17:57.011893 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:57.011859    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:57.012249 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:57.011908    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:17:57.012249 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:57.011987    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:57.012249 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:57.012029    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:57.012249 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:57.012042    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm podName:269dc00a-7bd4-4a18-97a3-2a1ec916dd93 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:05.012027662 +0000 UTC m=+81.269637331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zp6vz" (UID: "269dc00a-7bd4-4a18-97a3-2a1ec916dd93") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:57.012249 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:57.012095    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls podName:269dc00a-7bd4-4a18-97a3-2a1ec916dd93 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:05.012077528 +0000 UTC m=+81.269687195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zp6vz" (UID: "269dc00a-7bd4-4a18-97a3-2a1ec916dd93") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:59.632966 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:17:59.632926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-gd67w\" (UID: \"04d98107-270d-46aa-b9e4-013adb971e13\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:17:59.633404 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:59.633068    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:17:59.633404 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:17:59.633132    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls podName:04d98107-270d-46aa-b9e4-013adb971e13 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:15.633116934 +0000 UTC m=+91.890726601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-gd67w" (UID: "04d98107-270d-46aa-b9e4-013adb971e13") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:18:05.077409 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:05.077356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:18:05.077852 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:05.077432    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:18:05.077852 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:18:05.077511    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:18:05.077852 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:18:05.077600    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls podName:269dc00a-7bd4-4a18-97a3-2a1ec916dd93 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:21.077583036 +0000 UTC m=+97.335192705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zp6vz" (UID: "269dc00a-7bd4-4a18-97a3-2a1ec916dd93") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:18:05.078014 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:05.077995    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:18:07.306943 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:07.306818    2575 scope.go:117] "RemoveContainer" containerID="7ff450761ab748ca76ba7abb6c877ce163a2a5de7ca05e7ee3995b611f6165b1"
Jul 28 17:18:07.710508 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:07.710480    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:18:07.710667 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:07.710564    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-c679z" event={"ID":"7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7","Type":"ContainerStarted","Data":"4ad6c20d057fcd47c986dfd4a74264d14bfdcd67fbe7e7433a9c18132085b3bd"}
Jul 28 17:18:07.710847 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:07.710823    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:18:07.729116 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:07.729056    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-c679z" podStartSLOduration=21.21693195 podStartE2EDuration="24.729042036s" totalImagesPullingTime="3.512110086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:07.728212189 +0000 UTC m=+83.985821882" watchObservedRunningTime="2026-07-28 17:18:07.729042036 +0000 UTC m=+83.986651729"
Jul 28 17:18:08.376401 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:08.376371    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-c679z"
Jul 28 17:18:13.828462 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:13.828433    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6464d6977b-cl4xq"]
Jul 28 17:18:13.832598 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:13.831103    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6464d6977b-cl4xq"
Jul 28 17:18:13.834653 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:13.834629    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 17:18:13.835038 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:13.835014    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 17:18:13.835459 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:13.835439    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 17:18:13.835584 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:13.835481    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 17:18:13.835700 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:13.835688    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-n27lw\""
Jul 28 17:18:13.854565 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:13.854543    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6464d6977b-cl4xq"]
Jul 28 17:18:13.946449 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:13.946419    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f5c3f918-0091-45b4-9fcd-ea141a4dc024-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6464d6977b-cl4xq\" (UID: \"f5c3f918-0091-45b4-9fcd-ea141a4dc024\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6464d6977b-cl4xq"
Jul 28 17:18:13.946649 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:13.946471    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqcmk\" (UniqueName: \"kubernetes.io/projected/f5c3f918-0091-45b4-9fcd-ea141a4dc024-kube-api-access-sqcmk\") pod \"managed-serviceaccount-addon-agent-6464d6977b-cl4xq\" (UID: \"f5c3f918-0091-45b4-9fcd-ea141a4dc024\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6464d6977b-cl4xq"
Jul 28 17:18:14.047587 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:14.047563    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqcmk\" (UniqueName: \"kubernetes.io/projected/f5c3f918-0091-45b4-9fcd-ea141a4dc024-kube-api-access-sqcmk\") pod \"managed-serviceaccount-addon-agent-6464d6977b-cl4xq\" (UID: \"f5c3f918-0091-45b4-9fcd-ea141a4dc024\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6464d6977b-cl4xq"
Jul 28 17:18:14.047742 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:14.047645    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f5c3f918-0091-45b4-9fcd-ea141a4dc024-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6464d6977b-cl4xq\" (UID: \"f5c3f918-0091-45b4-9fcd-ea141a4dc024\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6464d6977b-cl4xq"
Jul 28 17:18:14.050147 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:14.050124    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f5c3f918-0091-45b4-9fcd-ea141a4dc024-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6464d6977b-cl4xq\" (UID: \"f5c3f918-0091-45b4-9fcd-ea141a4dc024\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6464d6977b-cl4xq"
Jul 28 17:18:14.067887 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:14.067858    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqcmk\" (UniqueName: \"kubernetes.io/projected/f5c3f918-0091-45b4-9fcd-ea141a4dc024-kube-api-access-sqcmk\") pod \"managed-serviceaccount-addon-agent-6464d6977b-cl4xq\" (UID: \"f5c3f918-0091-45b4-9fcd-ea141a4dc024\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6464d6977b-cl4xq"
Jul 28 17:18:14.149605 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:14.149517    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6464d6977b-cl4xq"
Jul 28 17:18:14.284169 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:14.284141    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6464d6977b-cl4xq"]
Jul 28 17:18:14.287217 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:18:14.287191    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5c3f918_0091_45b4_9fcd_ea141a4dc024.slice/crio-896c6881b7976c025882222d7eeb6b5cbf4b5122aa48a235f04314715230584d WatchSource:0}: Error finding container 896c6881b7976c025882222d7eeb6b5cbf4b5122aa48a235f04314715230584d: Status 404 returned error can't find the container with id 896c6881b7976c025882222d7eeb6b5cbf4b5122aa48a235f04314715230584d
Jul 28 17:18:14.731905 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:14.731871    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6464d6977b-cl4xq" event={"ID":"f5c3f918-0091-45b4-9fcd-ea141a4dc024","Type":"ContainerStarted","Data":"896c6881b7976c025882222d7eeb6b5cbf4b5122aa48a235f04314715230584d"}
Jul 28 17:18:15.662015 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:15.661978    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-gd67w\" (UID: \"04d98107-270d-46aa-b9e4-013adb971e13\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:18:15.664613 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:15.664576    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/04d98107-270d-46aa-b9e4-013adb971e13-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-gd67w\" (UID: \"04d98107-270d-46aa-b9e4-013adb971e13\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:18:15.962813 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:15.962745    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8lxpl\""
Jul 28 17:18:15.971461 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:15.971438    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"
Jul 28 17:18:16.928731 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:16.928702    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w"]
Jul 28 17:18:16.931616 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:18:16.931592    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d98107_270d_46aa_b9e4_013adb971e13.slice/crio-58241e731e826ea9501da57e1fc9b921429247025c0c3d8fb8ea56b95880adfb WatchSource:0}: Error finding container 58241e731e826ea9501da57e1fc9b921429247025c0c3d8fb8ea56b95880adfb: Status 404 returned error can't find the container with id 58241e731e826ea9501da57e1fc9b921429247025c0c3d8fb8ea56b95880adfb
Jul 28 17:18:17.744014 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:17.743970    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6464d6977b-cl4xq" event={"ID":"f5c3f918-0091-45b4-9fcd-ea141a4dc024","Type":"ContainerStarted","Data":"425839c7fdd0aa9efce16e7fc47d0118d09d61e4f540b10d707eeb32564e7b93"}
Jul 28 17:18:17.745228 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:17.745200    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w" event={"ID":"04d98107-270d-46aa-b9e4-013adb971e13","Type":"ContainerStarted","Data":"58241e731e826ea9501da57e1fc9b921429247025c0c3d8fb8ea56b95880adfb"}
Jul 28 17:18:17.759738 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:17.759689    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6464d6977b-cl4xq" podStartSLOduration=2.21273574 podStartE2EDuration="4.759674829s" totalImagesPullingTime="2.546939089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:17.758894656 +0000 UTC m=+94.016504348" watchObservedRunningTime="2026-07-28 17:18:17.759674829 +0000 UTC m=+94.017284518"
Jul 28 17:18:18.296805 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.296772    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b67c8b6c9-9crbf"]
Jul 28 17:18:18.303225 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.303188    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.305669 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.305645    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 17:18:18.305812 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.305795    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 17:18:18.305874 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.305795    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 17:18:18.305929 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.305872    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 17:18:18.306479 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.306460    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 17:18:18.306600 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.306568    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 17:18:18.306677 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.306661    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xc6lg\""
Jul 28 17:18:18.306763 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.306741    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 17:18:18.311659 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.311610    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b67c8b6c9-9crbf"]
Jul 28 17:18:18.385780 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.385743    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjxj7\" (UniqueName: \"kubernetes.io/projected/80ec7720-b034-4a97-9cc1-184f7b8ba541-kube-api-access-cjxj7\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.385970 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.385789    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-oauth-config\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.385970 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.385879    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-config\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.385970 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.385910    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-service-ca\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.385970 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.385950    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-oauth-serving-cert\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.386165 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.386010    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-serving-cert\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.486653 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.486611    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjxj7\" (UniqueName: \"kubernetes.io/projected/80ec7720-b034-4a97-9cc1-184f7b8ba541-kube-api-access-cjxj7\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.486827 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.486662    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-oauth-config\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.486827 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.486735    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-config\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.486827 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.486761    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-service-ca\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.486827 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.486804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-oauth-serving-cert\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.487038 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.486828    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-serving-cert\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.487569 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.487503    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-config\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.487569 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.487534    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-service-ca\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.487569 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.487559    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-oauth-serving-cert\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.489368 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.489349    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-oauth-config\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.489867 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.489848    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-serving-cert\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.495591 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.495572    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjxj7\" (UniqueName: \"kubernetes.io/projected/80ec7720-b034-4a97-9cc1-184f7b8ba541-kube-api-access-cjxj7\") pod \"console-6b67c8b6c9-9crbf\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") " pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.623932 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.623664    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:18.749783 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.749745    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w" event={"ID":"04d98107-270d-46aa-b9e4-013adb971e13","Type":"ContainerStarted","Data":"8d747c006e065bcedd3385183f1da3b881d857d96bf7519a0a32c05fd89d8d1d"}
Jul 28 17:18:18.756617 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.755897    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b67c8b6c9-9crbf"]
Jul 28 17:18:18.761620 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:18:18.761586    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ec7720_b034_4a97_9cc1_184f7b8ba541.slice/crio-9d0b44b682cb79b6ae5aaa6ebf129075252cacbe5f3c54bed60855a431b6e762 WatchSource:0}: Error finding container 9d0b44b682cb79b6ae5aaa6ebf129075252cacbe5f3c54bed60855a431b6e762: Status 404 returned error can't find the container with id 9d0b44b682cb79b6ae5aaa6ebf129075252cacbe5f3c54bed60855a431b6e762
Jul 28 17:18:18.772391 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:18.772344    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-gd67w" podStartSLOduration=34.117084851 podStartE2EDuration="35.772326094s" totalImagesPullingTime="1.655241243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:18.77122287 +0000 UTC m=+95.028832561" watchObservedRunningTime="2026-07-28 17:18:18.772326094 +0000 UTC m=+95.029935785"
Jul 28 17:18:19.164268 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:19.164234    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx"]
Jul 28 17:18:19.166565 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:19.166548    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx"
Jul 28 17:18:19.168588 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:19.168566    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 17:18:19.168803 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:19.168790    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jvtsd\""
Jul 28 17:18:19.175789 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:19.175767    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx"]
Jul 28 17:18:19.292712 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:19.292674    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40036ef4-7b56-419d-83e5-618fdd5121ab-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-q6vnx\" (UID: \"40036ef4-7b56-419d-83e5-618fdd5121ab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx"
Jul 28 17:18:19.393660 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:19.393624    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40036ef4-7b56-419d-83e5-618fdd5121ab-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-q6vnx\" (UID: \"40036ef4-7b56-419d-83e5-618fdd5121ab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx"
Jul 28 17:18:19.394038 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:18:19.393775    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 17:18:19.394038 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:18:19.393843    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40036ef4-7b56-419d-83e5-618fdd5121ab-tls-certificates podName:40036ef4-7b56-419d-83e5-618fdd5121ab nodeName:}" failed. No retries permitted until 2026-07-28 17:18:19.893826432 +0000 UTC m=+96.151436104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/40036ef4-7b56-419d-83e5-618fdd5121ab-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-q6vnx" (UID: "40036ef4-7b56-419d-83e5-618fdd5121ab") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 17:18:19.755109 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:19.755026    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b67c8b6c9-9crbf" event={"ID":"80ec7720-b034-4a97-9cc1-184f7b8ba541","Type":"ContainerStarted","Data":"9d0b44b682cb79b6ae5aaa6ebf129075252cacbe5f3c54bed60855a431b6e762"}
Jul 28 17:18:19.897656 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:19.897615    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40036ef4-7b56-419d-83e5-618fdd5121ab-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-q6vnx\" (UID: \"40036ef4-7b56-419d-83e5-618fdd5121ab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx"
Jul 28 17:18:19.900642 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:19.900613    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40036ef4-7b56-419d-83e5-618fdd5121ab-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-q6vnx\" (UID: \"40036ef4-7b56-419d-83e5-618fdd5121ab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx"
Jul 28 17:18:20.075778 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:20.075692    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx"
Jul 28 17:18:20.228269 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:20.228236    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx"]
Jul 28 17:18:20.232312 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:18:20.232284    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40036ef4_7b56_419d_83e5_618fdd5121ab.slice/crio-90b631d6045aa800181de5d55c3689af5400eb4463386f4d77bfb1f6d4b448c7 WatchSource:0}: Error finding container 90b631d6045aa800181de5d55c3689af5400eb4463386f4d77bfb1f6d4b448c7: Status 404 returned error can't find the container with id 90b631d6045aa800181de5d55c3689af5400eb4463386f4d77bfb1f6d4b448c7
Jul 28 17:18:20.657614 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:20.657582    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n6cbg"
Jul 28 17:18:20.759317 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:20.759279    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx" event={"ID":"40036ef4-7b56-419d-83e5-618fdd5121ab","Type":"ContainerStarted","Data":"90b631d6045aa800181de5d55c3689af5400eb4463386f4d77bfb1f6d4b448c7"}
Jul 28 17:18:21.109873 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:21.109771    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:18:21.112566 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:21.112541    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269dc00a-7bd4-4a18-97a3-2a1ec916dd93-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zp6vz\" (UID: \"269dc00a-7bd4-4a18-97a3-2a1ec916dd93\") " pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:18:21.126043 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:21.126021    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zp6vz"
Jul 28 17:18:22.828912 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:22.828883    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zp6vz"]
Jul 28 17:18:22.833017 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:18:22.832987    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269dc00a_7bd4_4a18_97a3_2a1ec916dd93.slice/crio-486d1a76460b4d26f798f5a7b6d28a8ba962da67e3217fe27021e657a16ab74e WatchSource:0}: Error finding container 486d1a76460b4d26f798f5a7b6d28a8ba962da67e3217fe27021e657a16ab74e: Status 404 returned error can't find the container with id 486d1a76460b4d26f798f5a7b6d28a8ba962da67e3217fe27021e657a16ab74e
Jul 28 17:18:23.528339 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.527901    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:18:23.531210 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.531162    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls\") pod \"image-registry-dfbf9859d-q5c99\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") " pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:18:23.546542 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.546335    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gt8nm\""
Jul 28 17:18:23.554594 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.554505    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:18:23.628943 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.628910    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:18:23.636569 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.636492    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56add391-4f0b-4bf8-a857-563b82d92fbf-metrics-tls\") pod \"dns-default-2nw7c\" (UID: \"56add391-4f0b-4bf8-a857-563b82d92fbf\") " pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:18:23.734474 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.734394    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dfbf9859d-q5c99"]
Jul 28 17:18:23.738837 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:18:23.738811    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15ef051_198a_49eb_b21d_1d8a7b82ccba.slice/crio-61862394a2bbcf0969dea990957d15ede4dca4c1d668bf46f9a45b2162d9b7f6 WatchSource:0}: Error finding container 61862394a2bbcf0969dea990957d15ede4dca4c1d668bf46f9a45b2162d9b7f6: Status 404 returned error can't find the container with id 61862394a2bbcf0969dea990957d15ede4dca4c1d668bf46f9a45b2162d9b7f6
Jul 28 17:18:23.771993 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.771912    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dfbf9859d-q5c99" event={"ID":"a15ef051-198a-49eb-b21d-1d8a7b82ccba","Type":"ContainerStarted","Data":"61862394a2bbcf0969dea990957d15ede4dca4c1d668bf46f9a45b2162d9b7f6"}
Jul 28 17:18:23.773426 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.773383    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zp6vz" event={"ID":"269dc00a-7bd4-4a18-97a3-2a1ec916dd93","Type":"ContainerStarted","Data":"486d1a76460b4d26f798f5a7b6d28a8ba962da67e3217fe27021e657a16ab74e"}
Jul 28 17:18:23.776108 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.776034    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b67c8b6c9-9crbf" event={"ID":"80ec7720-b034-4a97-9cc1-184f7b8ba541","Type":"ContainerStarted","Data":"0a5f4e3cc63b156f840cedf9cf763a99221cca879e47b26af46e337952a36b18"}
Jul 28 17:18:23.778544 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.778471    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx" event={"ID":"40036ef4-7b56-419d-83e5-618fdd5121ab","Type":"ContainerStarted","Data":"9fa85156c74c939973d6ab1f4468ec72b221388ed8ab913d81dca0caebe86b66"}
Jul 28 17:18:23.779306 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.779221    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx"
Jul 28 17:18:23.785133 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.785088    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx"
Jul 28 17:18:23.794861 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.794735    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b67c8b6c9-9crbf" podStartSLOduration=1.852689798 podStartE2EDuration="5.794716371s" totalImagesPullingTime="3.942026573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:23.793730794 +0000 UTC m=+100.051340487" watchObservedRunningTime="2026-07-28 17:18:23.794716371 +0000 UTC m=+100.052326062"
Jul 28 17:18:23.809934 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.809886    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6vnx" podStartSLOduration=2.337474941 podStartE2EDuration="4.809865697s" totalImagesPullingTime="2.472390756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:23.807998505 +0000 UTC m=+100.065608197" watchObservedRunningTime="2026-07-28 17:18:23.809865697 +0000 UTC m=+100.067475390"
Jul 28 17:18:23.873461 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.873436    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c9wr5\""
Jul 28 17:18:23.881834 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:23.881808    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:18:24.044366 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.044281    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2nw7c"]
Jul 28 17:18:24.047392 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:18:24.047363    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56add391_4f0b_4bf8_a857_563b82d92fbf.slice/crio-737a8c644dd1bbe4ce6d8c50852ec50e3f645901f11dd550d83205d9462d2b3e WatchSource:0}: Error finding container 737a8c644dd1bbe4ce6d8c50852ec50e3f645901f11dd550d83205d9462d2b3e: Status 404 returned error can't find the container with id 737a8c644dd1bbe4ce6d8c50852ec50e3f645901f11dd550d83205d9462d2b3e
Jul 28 17:18:24.220687 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.220478    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"]
Jul 28 17:18:24.242964 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.242936    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"]
Jul 28 17:18:24.243138 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.243058    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"
Jul 28 17:18:24.245709 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.245658    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 17:18:24.245935 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.245913    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:18:24.246075 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.246006    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 17:18:24.246151 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.246083    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-j4fs5\""
Jul 28 17:18:24.336177 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.336087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ng7mq\" (UID: \"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"
Jul 28 17:18:24.336177 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.336149    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-ng7mq\" (UID: \"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"
Jul 28 17:18:24.336384 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.336359    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-ng7mq\" (UID: \"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"
Jul 28 17:18:24.336437 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.336398    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czrbt\" (UniqueName: \"kubernetes.io/projected/74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2-kube-api-access-czrbt\") pod \"prometheus-operator-84d69f794d-ng7mq\" (UID: \"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"
Jul 28 17:18:24.437030 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.436992    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-ng7mq\" (UID: \"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"
Jul 28 17:18:24.437203 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.437039    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czrbt\" (UniqueName: \"kubernetes.io/projected/74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2-kube-api-access-czrbt\") pod \"prometheus-operator-84d69f794d-ng7mq\" (UID: \"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"
Jul 28 17:18:24.437203 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.437101    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ng7mq\" (UID: \"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"
Jul 28 17:18:24.437203 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.437141    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-ng7mq\" (UID: \"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"
Jul 28 17:18:24.437350 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:18:24.437244    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 17:18:24.437350 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:18:24.437328    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2-prometheus-operator-tls podName:74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:24.937306122 +0000 UTC m=+101.194915794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-ng7mq" (UID: "74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2") : secret "prometheus-operator-tls" not found
Jul 28 17:18:24.438056 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.438031    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-ng7mq\" (UID: \"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"
Jul 28 17:18:24.439614 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.439594    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-ng7mq\" (UID: \"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"
Jul 28 17:18:24.446052 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.446025    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czrbt\" (UniqueName: \"kubernetes.io/projected/74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2-kube-api-access-czrbt\") pod \"prometheus-operator-84d69f794d-ng7mq\" (UID: \"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"
Jul 28 17:18:24.784667 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.784627    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dfbf9859d-q5c99" event={"ID":"a15ef051-198a-49eb-b21d-1d8a7b82ccba","Type":"ContainerStarted","Data":"315cbf9e7db50d9be04eaa735ce1eb5b24a6c9918f222216bf7ff855add1e78f"}
Jul 28 17:18:24.784847 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.784732    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:18:24.786181 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.785864    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2nw7c" event={"ID":"56add391-4f0b-4bf8-a857-563b82d92fbf","Type":"ContainerStarted","Data":"737a8c644dd1bbe4ce6d8c50852ec50e3f645901f11dd550d83205d9462d2b3e"}
Jul 28 17:18:24.807687 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.807592    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-dfbf9859d-q5c99" podStartSLOduration=99.807573618 podStartE2EDuration="1m39.807573618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:24.806138237 +0000 UTC m=+101.063747923" watchObservedRunningTime="2026-07-28 17:18:24.807573618 +0000 UTC m=+101.065183307"
Jul 28 17:18:24.943306 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.943256    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ng7mq\" (UID: \"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"
Jul 28 17:18:24.946976 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:24.946950    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ng7mq\" (UID: \"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"
Jul 28 17:18:25.154998 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:25.154915    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"
Jul 28 17:18:25.316511 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:25.316480    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-ng7mq"]
Jul 28 17:18:25.716731 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:18:25.716692    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a15adc_a35a_49c4_a2fe_ccf52fbe2ea2.slice/crio-80a2b76fb0f1b1889044c6b965aa73c26cb52b6790125698a278216f857ab2ff WatchSource:0}: Error finding container 80a2b76fb0f1b1889044c6b965aa73c26cb52b6790125698a278216f857ab2ff: Status 404 returned error can't find the container with id 80a2b76fb0f1b1889044c6b965aa73c26cb52b6790125698a278216f857ab2ff
Jul 28 17:18:25.791270 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:25.791228    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq" event={"ID":"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2","Type":"ContainerStarted","Data":"80a2b76fb0f1b1889044c6b965aa73c26cb52b6790125698a278216f857ab2ff"}
Jul 28 17:18:28.624994 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:28.624950    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:28.625484 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:28.625021    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:28.630632 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:28.630608    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:28.804165 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:28.804098    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zp6vz" event={"ID":"269dc00a-7bd4-4a18-97a3-2a1ec916dd93","Type":"ContainerStarted","Data":"8d7e7c979666ba0b5d64c64172b31ccdae19ed9b29e34ffe3d7815209abfcb06"}
Jul 28 17:18:28.807033 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:28.807002    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq" event={"ID":"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2","Type":"ContainerStarted","Data":"bbe66c0bf96e04dc500853b9145f5324aa2073f61d70f1c64145c4ce098ff850"}
Jul 28 17:18:28.813333 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:28.813305    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2nw7c" event={"ID":"56add391-4f0b-4bf8-a857-563b82d92fbf","Type":"ContainerStarted","Data":"ffd25718204d299c8b6a774a5d1f6faf32c040f6b252455a13b4a128f45ee3dd"}
Jul 28 17:18:28.818948 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:28.818844    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:18:29.815626 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:29.815582    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq" event={"ID":"74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2","Type":"ContainerStarted","Data":"2defc9eb9b6b429bfe9c7d5305266c205cac9579de895e80382ad036c40eb2a9"}
Jul 28 17:18:29.817565 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:29.817535    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2nw7c" event={"ID":"56add391-4f0b-4bf8-a857-563b82d92fbf","Type":"ContainerStarted","Data":"593a301126c5b7af6c9366c2cc0612ecbba2f62c915707b220b618e58217fe25"}
Jul 28 17:18:29.835253 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:29.835206    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-ng7mq" podStartSLOduration=2.815717641 podStartE2EDuration="5.835191408s" totalImagesPullingTime="3.019473767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:29.834535998 +0000 UTC m=+106.092145690" watchObservedRunningTime="2026-07-28 17:18:29.835191408 +0000 UTC m=+106.092801098"
Jul 28 17:18:29.854760 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:29.854704    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2nw7c" podStartSLOduration=66.521574527 podStartE2EDuration="1m10.854683519s" totalImagesPullingTime="4.333108992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:29.85368186 +0000 UTC m=+106.111291552" watchObservedRunningTime="2026-07-28 17:18:29.854683519 +0000 UTC m=+106.112293214"
Jul 28 17:18:30.821635 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:30.821601    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:18:31.613250 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.613217    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-66zdn"]
Jul 28 17:18:31.620928 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.620902    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.623178 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.623151    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kzkgt\""
Jul 28 17:18:31.623317 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.623216    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:18:31.623317 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.623252    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:18:31.623432 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.623413    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:18:31.634021 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.633995    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"]
Jul 28 17:18:31.636532 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.636500    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.639706 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.639684    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 17:18:31.639805 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.639708    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 17:18:31.639805 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.639714    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-drrzl\""
Jul 28 17:18:31.640730 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.640710    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 17:18:31.665408 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.665381    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"]
Jul 28 17:18:31.809009 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.808977    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/47211b67-018e-4ffb-9a28-081195f36db3-node-exporter-textfile\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.809178 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.809016    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47211b67-018e-4ffb-9a28-081195f36db3-sys\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.809178 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.809042    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/88fd4d31-43e6-41f6-984e-abec4a426779-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.809178 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.809129    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88fd4d31-43e6-41f6-984e-abec4a426779-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.809178 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.809168    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d89q4\" (UniqueName: \"kubernetes.io/projected/47211b67-018e-4ffb-9a28-081195f36db3-kube-api-access-d89q4\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.809353 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.809248    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88fd4d31-43e6-41f6-984e-abec4a426779-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.809353 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.809269    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47211b67-018e-4ffb-9a28-081195f36db3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.809353 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.809287    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/47211b67-018e-4ffb-9a28-081195f36db3-node-exporter-tls\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.809353 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.809332    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/88fd4d31-43e6-41f6-984e-abec4a426779-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.809536 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.809376    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/47211b67-018e-4ffb-9a28-081195f36db3-root\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.809536 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.809399    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/47211b67-018e-4ffb-9a28-081195f36db3-node-exporter-accelerators-collector-config\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.809536 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.809416    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t825\" (UniqueName: \"kubernetes.io/projected/88fd4d31-43e6-41f6-984e-abec4a426779-kube-api-access-5t825\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.809536 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.809445    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/47211b67-018e-4ffb-9a28-081195f36db3-node-exporter-wtmp\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.809536 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.809468    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47211b67-018e-4ffb-9a28-081195f36db3-metrics-client-ca\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.809536 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.809509    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88fd4d31-43e6-41f6-984e-abec4a426779-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.910707 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.910680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47211b67-018e-4ffb-9a28-081195f36db3-metrics-client-ca\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.911147 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.910727    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88fd4d31-43e6-41f6-984e-abec4a426779-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.911147 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.910778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/47211b67-018e-4ffb-9a28-081195f36db3-node-exporter-textfile\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.911147 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.910814    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47211b67-018e-4ffb-9a28-081195f36db3-sys\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.911147 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.910844    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/88fd4d31-43e6-41f6-984e-abec4a426779-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.911147 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.910879    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88fd4d31-43e6-41f6-984e-abec4a426779-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.911147 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.910902    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d89q4\" (UniqueName: \"kubernetes.io/projected/47211b67-018e-4ffb-9a28-081195f36db3-kube-api-access-d89q4\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.911147 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.910941    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47211b67-018e-4ffb-9a28-081195f36db3-sys\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.911147 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.910948    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88fd4d31-43e6-41f6-984e-abec4a426779-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.911147 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.911008    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47211b67-018e-4ffb-9a28-081195f36db3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.911147 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.911039    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/47211b67-018e-4ffb-9a28-081195f36db3-node-exporter-tls\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.911147 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.911081    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/88fd4d31-43e6-41f6-984e-abec4a426779-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.911147 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.911104    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/47211b67-018e-4ffb-9a28-081195f36db3-node-exporter-textfile\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.911147 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.911116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/47211b67-018e-4ffb-9a28-081195f36db3-root\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.912034 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.911162    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/47211b67-018e-4ffb-9a28-081195f36db3-node-exporter-accelerators-collector-config\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.912034 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.911165    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/47211b67-018e-4ffb-9a28-081195f36db3-root\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.912034 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.911202    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t825\" (UniqueName: \"kubernetes.io/projected/88fd4d31-43e6-41f6-984e-abec4a426779-kube-api-access-5t825\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.912034 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.911257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/47211b67-018e-4ffb-9a28-081195f36db3-node-exporter-wtmp\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.912034 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.911279    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/88fd4d31-43e6-41f6-984e-abec4a426779-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.912034 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.911286    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47211b67-018e-4ffb-9a28-081195f36db3-metrics-client-ca\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.912034 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.911737    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/47211b67-018e-4ffb-9a28-081195f36db3-node-exporter-wtmp\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.912371 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.912193    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/47211b67-018e-4ffb-9a28-081195f36db3-node-exporter-accelerators-collector-config\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.912424 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.912379    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/88fd4d31-43e6-41f6-984e-abec4a426779-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.912823 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.912797    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88fd4d31-43e6-41f6-984e-abec4a426779-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.913512 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.913495    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/47211b67-018e-4ffb-9a28-081195f36db3-node-exporter-tls\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.913661 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.913640    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88fd4d31-43e6-41f6-984e-abec4a426779-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.913732 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.913719    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88fd4d31-43e6-41f6-984e-abec4a426779-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.914280 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.914265    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47211b67-018e-4ffb-9a28-081195f36db3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.923279 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.923259    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d89q4\" (UniqueName: \"kubernetes.io/projected/47211b67-018e-4ffb-9a28-081195f36db3-kube-api-access-d89q4\") pod \"node-exporter-66zdn\" (UID: \"47211b67-018e-4ffb-9a28-081195f36db3\") " pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.924884 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.924863    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t825\" (UniqueName: \"kubernetes.io/projected/88fd4d31-43e6-41f6-984e-abec4a426779-kube-api-access-5t825\") pod \"kube-state-metrics-56d9f944b4-vvvrp\" (UID: \"88fd4d31-43e6-41f6-984e-abec4a426779\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:31.931697 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.931672    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-66zdn"
Jul 28 17:18:31.946602 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:31.946584    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"
Jul 28 17:18:32.016616 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:18:32.016584    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47211b67_018e_4ffb_9a28_081195f36db3.slice/crio-f571280d4f4b0ed6b3837555f012bbdbdc9ce9ca6e27f28a6122872e59c1b535 WatchSource:0}: Error finding container f571280d4f4b0ed6b3837555f012bbdbdc9ce9ca6e27f28a6122872e59c1b535: Status 404 returned error can't find the container with id f571280d4f4b0ed6b3837555f012bbdbdc9ce9ca6e27f28a6122872e59c1b535
Jul 28 17:18:32.141161 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:32.141129    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp"]
Jul 28 17:18:32.144398 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:18:32.144371    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88fd4d31_43e6_41f6_984e_abec4a426779.slice/crio-0ef39a167992dc84bded7fdfbd741e5e47e5f4733ad7ecabae6b7261d36a4903 WatchSource:0}: Error finding container 0ef39a167992dc84bded7fdfbd741e5e47e5f4733ad7ecabae6b7261d36a4903: Status 404 returned error can't find the container with id 0ef39a167992dc84bded7fdfbd741e5e47e5f4733ad7ecabae6b7261d36a4903
Jul 28 17:18:32.834836 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:32.834743    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66zdn" event={"ID":"47211b67-018e-4ffb-9a28-081195f36db3","Type":"ContainerStarted","Data":"f571280d4f4b0ed6b3837555f012bbdbdc9ce9ca6e27f28a6122872e59c1b535"}
Jul 28 17:18:32.837176 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:32.837091    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp" event={"ID":"88fd4d31-43e6-41f6-984e-abec4a426779","Type":"ContainerStarted","Data":"0ef39a167992dc84bded7fdfbd741e5e47e5f4733ad7ecabae6b7261d36a4903"}
Jul 28 17:18:32.843598 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:32.843538    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zp6vz" event={"ID":"269dc00a-7bd4-4a18-97a3-2a1ec916dd93","Type":"ContainerStarted","Data":"86609884eafa5f3704fb7950d366ffbd52af5bfa57774f9016a4abd071533509"}
Jul 28 17:18:33.852648 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:33.852578    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66zdn" event={"ID":"47211b67-018e-4ffb-9a28-081195f36db3","Type":"ContainerStarted","Data":"b18037bd2e02e54bf5c94da96f9a71c894c580a78c2453ddf78e360a98d8208d"}
Jul 28 17:18:33.855067 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:33.854994    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp" event={"ID":"88fd4d31-43e6-41f6-984e-abec4a426779","Type":"ContainerStarted","Data":"49c3566db7467685b055a12920c2500119df6af7deacb2ede2beef709c0d5413"}
Jul 28 17:18:34.861274 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:34.861237    2575 generic.go:356] "Generic (PLEG): container finished" podID="47211b67-018e-4ffb-9a28-081195f36db3" containerID="b18037bd2e02e54bf5c94da96f9a71c894c580a78c2453ddf78e360a98d8208d" exitCode=0
Jul 28 17:18:34.861717 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:34.861329    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66zdn" event={"ID":"47211b67-018e-4ffb-9a28-081195f36db3","Type":"ContainerDied","Data":"b18037bd2e02e54bf5c94da96f9a71c894c580a78c2453ddf78e360a98d8208d"}
Jul 28 17:18:34.863750 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:34.863728    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp" event={"ID":"88fd4d31-43e6-41f6-984e-abec4a426779","Type":"ContainerStarted","Data":"366f87d55f3a06831f42954b667781e4ad77e30777723d45dc6674927c7c34aa"}
Jul 28 17:18:34.863869 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:34.863777    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp" event={"ID":"88fd4d31-43e6-41f6-984e-abec4a426779","Type":"ContainerStarted","Data":"9cd5cd90abb3f3a372cdb1d2d250ce1cdeb58195192ea41e33ba4b0432d79c28"}
Jul 28 17:18:34.900540 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:34.900474    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvvrp" podStartSLOduration=2.3262372510000002 podStartE2EDuration="3.900458417s" totalImagesPullingTime="1.574221166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:34.89932449 +0000 UTC m=+111.156934182" watchObservedRunningTime="2026-07-28 17:18:34.900458417 +0000 UTC m=+111.158068110"
Jul 28 17:18:35.637131 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:35.637091    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dfbf9859d-q5c99"]
Jul 28 17:18:36.426379 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.426346    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dbff64df6-8bftp"]
Jul 28 17:18:36.428424 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.428403    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.443101 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.443079    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 17:18:36.453751 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.453729    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hfs9\" (UniqueName: \"kubernetes.io/projected/981b07f5-ecb6-4d96-a63d-9575da136d8a-kube-api-access-5hfs9\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.453847 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.453760    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-service-ca\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.453847 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.453796    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-oauth-config\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.453913 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.453865    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-trusted-ca-bundle\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.453913 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.453891    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-oauth-serving-cert\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.453989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.453914    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-config\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.453989 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.453934    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-serving-cert\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.459777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.459752    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dbff64df6-8bftp"]
Jul 28 17:18:36.555087 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.555052    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-trusted-ca-bundle\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.555087 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.555088    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-oauth-serving-cert\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.555309 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.555109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-config\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.555309 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.555144    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-serving-cert\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.555309 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.555174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hfs9\" (UniqueName: \"kubernetes.io/projected/981b07f5-ecb6-4d96-a63d-9575da136d8a-kube-api-access-5hfs9\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.555309 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.555203    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-service-ca\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.555309 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.555272    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-oauth-config\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.555898 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.555868    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-oauth-serving-cert\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.556028 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.555920    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-config\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.556028 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.555967    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-service-ca\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.556326 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.556305    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-trusted-ca-bundle\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.557742 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.557720    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-oauth-config\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.557933 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.557913    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-serving-cert\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.567801 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.567776    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hfs9\" (UniqueName: \"kubernetes.io/projected/981b07f5-ecb6-4d96-a63d-9575da136d8a-kube-api-access-5hfs9\") pod \"console-5dbff64df6-8bftp\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") " pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.737836 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.737744    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:36.869694 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.869616    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dbff64df6-8bftp"]
Jul 28 17:18:36.872983 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:18:36.872952    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981b07f5_ecb6_4d96_a63d_9575da136d8a.slice/crio-45ed638c1c2a8e412faa714921521b74f286522b984992abbf2bd12aa4dafa4d WatchSource:0}: Error finding container 45ed638c1c2a8e412faa714921521b74f286522b984992abbf2bd12aa4dafa4d: Status 404 returned error can't find the container with id 45ed638c1c2a8e412faa714921521b74f286522b984992abbf2bd12aa4dafa4d
Jul 28 17:18:36.875172 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.875147    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66zdn" event={"ID":"47211b67-018e-4ffb-9a28-081195f36db3","Type":"ContainerStarted","Data":"ae45b2f163c189844748e57d2780606ff323a1979ce7d36bcb316677f8cfb6ca"}
Jul 28 17:18:36.875267 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.875208    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66zdn" event={"ID":"47211b67-018e-4ffb-9a28-081195f36db3","Type":"ContainerStarted","Data":"ac750aa86ae87d56847406bf1f8d7eba9cd3623324a0a0041ee3bf51cf6ec166"}
Jul 28 17:18:36.879162 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.879091    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zp6vz" event={"ID":"269dc00a-7bd4-4a18-97a3-2a1ec916dd93","Type":"ContainerStarted","Data":"3ce5629b3cd8c7d2e530783c2b7b3d9a40f605d9876c648598264fd9e6af2c4d"}
Jul 28 17:18:36.900084 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.900016    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-66zdn" podStartSLOduration=4.091075733 podStartE2EDuration="5.90000028s" totalImagesPullingTime="1.685612812s" totalInitContainerRuntime="123.311735ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:36.898794318 +0000 UTC m=+113.156403999" watchObservedRunningTime="2026-07-28 17:18:36.90000028 +0000 UTC m=+113.157609970"
Jul 28 17:18:36.918064 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:36.917994    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zp6vz" podStartSLOduration=34.758184567 podStartE2EDuration="47.91797787s" totalImagesPullingTime="13.159793303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:36.917375782 +0000 UTC m=+113.174985472" watchObservedRunningTime="2026-07-28 17:18:36.91797787 +0000 UTC m=+113.175587596"
Jul 28 17:18:37.884337 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.884295    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dbff64df6-8bftp" event={"ID":"981b07f5-ecb6-4d96-a63d-9575da136d8a","Type":"ContainerStarted","Data":"f6fcceb8b83da5bc3e7025b7025d03cbe1356d382f505c1d819feee6db0c82a0"}
Jul 28 17:18:37.884844 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.884357    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dbff64df6-8bftp" event={"ID":"981b07f5-ecb6-4d96-a63d-9575da136d8a","Type":"ContainerStarted","Data":"45ed638c1c2a8e412faa714921521b74f286522b984992abbf2bd12aa4dafa4d"}
Jul 28 17:18:37.909593 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.909226    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5dbff64df6-8bftp" podStartSLOduration=1.909204607 podStartE2EDuration="1.909204607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:37.908925093 +0000 UTC m=+114.166534784" watchObservedRunningTime="2026-07-28 17:18:37.909204607 +0000 UTC m=+114.166814300"
Jul 28 17:18:37.937926 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.937886    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:18:37.941954 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.941927    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.944207 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.944184    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 17:18:37.944601 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.944570    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 17:18:37.944713 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.944607    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 17:18:37.944713 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.944574    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 17:18:37.944713 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.944577    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 17:18:37.944902 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.944858    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 17:18:37.944971 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.944906    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 17:18:37.945025 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.944992    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 17:18:37.945339 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.945318    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 17:18:37.945440 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.945344    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 17:18:37.945503 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.945439    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 17:18:37.945503 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.945468    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 17:18:37.945746 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.945729    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d5g2c\""
Jul 28 17:18:37.945809 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.945736    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 17:18:37.945809 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.945791    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3a6675il53gf2\""
Jul 28 17:18:37.945912 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.945852    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 17:18:37.949701 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.949674    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 17:18:37.960024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.959994    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:18:37.965402 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.965373    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-config\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.965557 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.965402    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e484d34c-a1f2-4c8c-8df3-5e517676ef79-config-out\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.965629 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.965549    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.965629 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.965612    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e484d34c-a1f2-4c8c-8df3-5e517676ef79-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.965742 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.965656    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.965742 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.965680    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46fsr\" (UniqueName: \"kubernetes.io/projected/e484d34c-a1f2-4c8c-8df3-5e517676ef79-kube-api-access-46fsr\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.965742 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.965722    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.965892 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.965844    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.965951 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.965894    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e484d34c-a1f2-4c8c-8df3-5e517676ef79-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.966006 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.965972    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.966062 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.966005    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.966062 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.966039    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.966337 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.966249    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.966337 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.966293    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.966337 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.966328    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.966565 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.966412    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.966565 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.966433    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.966565 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.966501    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-web-config\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.966565 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.966541    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:37.966565 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:37.966556    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.067815 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.067783    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.067815 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.067822    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e484d34c-a1f2-4c8c-8df3-5e517676ef79-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068063 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.067853    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068063 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.067957    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068063 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.067986    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068222 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068063    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068222 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068103    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068222 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068130    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068222 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068201    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068414 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068234    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068414 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068295    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e484d34c-a1f2-4c8c-8df3-5e517676ef79-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068510 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068401    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-web-config\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068510 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068450    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068510 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068492    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068682 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068548    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-config\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068682 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e484d34c-a1f2-4c8c-8df3-5e517676ef79-config-out\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068682 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068610    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068682 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068637    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e484d34c-a1f2-4c8c-8df3-5e517676ef79-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068706    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46fsr\" (UniqueName: \"kubernetes.io/projected/e484d34c-a1f2-4c8c-8df3-5e517676ef79-kube-api-access-46fsr\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068733    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068760    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068817    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.068877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.068817    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.069178 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.069153    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.071264 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.070291    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.071264 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.071092    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.071264 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.071095    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.074877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.072232    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.074877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.072833    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.074877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.073217    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-web-config\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.074877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.073515    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.074877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.073556    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-config\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.074877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.073634    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e484d34c-a1f2-4c8c-8df3-5e517676ef79-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.074877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.073708    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e484d34c-a1f2-4c8c-8df3-5e517676ef79-config-out\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.074877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.073911    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.074877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.074206    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.074877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.074836    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e484d34c-a1f2-4c8c-8df3-5e517676ef79-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.076104 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.076081    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e484d34c-a1f2-4c8c-8df3-5e517676ef79-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.079207 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.079185    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46fsr\" (UniqueName: \"kubernetes.io/projected/e484d34c-a1f2-4c8c-8df3-5e517676ef79-kube-api-access-46fsr\") pod \"prometheus-k8s-0\" (UID: \"e484d34c-a1f2-4c8c-8df3-5e517676ef79\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.260678 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.260640    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:38.442792 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.442622    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:18:38.445473 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:18:38.445452    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode484d34c_a1f2_4c8c_8df3_5e517676ef79.slice/crio-9350ca167cb513bf98a8182c7babcaddac9271a7aad382c0a3e0a8d5b1287d9b WatchSource:0}: Error finding container 9350ca167cb513bf98a8182c7babcaddac9271a7aad382c0a3e0a8d5b1287d9b: Status 404 returned error can't find the container with id 9350ca167cb513bf98a8182c7babcaddac9271a7aad382c0a3e0a8d5b1287d9b
Jul 28 17:18:38.891053 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:38.891012    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e484d34c-a1f2-4c8c-8df3-5e517676ef79","Type":"ContainerStarted","Data":"9350ca167cb513bf98a8182c7babcaddac9271a7aad382c0a3e0a8d5b1287d9b"}
Jul 28 17:18:39.897254 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:39.897215    2575 generic.go:356] "Generic (PLEG): container finished" podID="e484d34c-a1f2-4c8c-8df3-5e517676ef79" containerID="b15385d5497ac5350da84faac74c02608e0a23a6287cc91c09454c8d0f1448c3" exitCode=0
Jul 28 17:18:39.897739 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:39.897299    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e484d34c-a1f2-4c8c-8df3-5e517676ef79","Type":"ContainerDied","Data":"b15385d5497ac5350da84faac74c02608e0a23a6287cc91c09454c8d0f1448c3"}
Jul 28 17:18:40.827683 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:40.827646    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2nw7c"
Jul 28 17:18:42.919852 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:42.919821    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e484d34c-a1f2-4c8c-8df3-5e517676ef79","Type":"ContainerStarted","Data":"3848d69fbe38eb81980f16e9d474a20449a008f0144e8f17acc863be9d680fd0"}
Jul 28 17:18:43.926542 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:43.926489    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e484d34c-a1f2-4c8c-8df3-5e517676ef79","Type":"ContainerStarted","Data":"c30d9a8ad566446a8b95b9123094d8158673042a0a59a035f8882204bda511ed"}
Jul 28 17:18:44.933754 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:44.933718    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e484d34c-a1f2-4c8c-8df3-5e517676ef79","Type":"ContainerStarted","Data":"879ad194b7c7401c7264b70a351a124056f4dcd9b83d1b6626dd2f5e02a229bd"}
Jul 28 17:18:44.934152 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:44.933812    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e484d34c-a1f2-4c8c-8df3-5e517676ef79","Type":"ContainerStarted","Data":"a96c4eecbe279cb3c03474c169b194b70da58f5ecb2f35c91bdf314e20203d1b"}
Jul 28 17:18:44.934152 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:44.933844    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e484d34c-a1f2-4c8c-8df3-5e517676ef79","Type":"ContainerStarted","Data":"b20eb296bacbd0fbc1d4df6ffa71c76e8ea518f88e97811182726aff24f6b826"}
Jul 28 17:18:44.934152 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:44.933869    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e484d34c-a1f2-4c8c-8df3-5e517676ef79","Type":"ContainerStarted","Data":"faa0859194cdd81b05283779cd5523684abd37ad0b95cf90babc7407c90ca9c1"}
Jul 28 17:18:44.964478 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:44.964423    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.643258737 podStartE2EDuration="7.964405686s" totalImagesPullingTime="5.302964296s" totalInitContainerRuntime="18.182653ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:44.963011283 +0000 UTC m=+121.220620976" watchObservedRunningTime="2026-07-28 17:18:44.964405686 +0000 UTC m=+121.222015377"
Jul 28 17:18:45.642612 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:45.642585    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:18:46.738456 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:46.738427    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:46.738867 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:46.738514    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:46.743899 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:46.743878    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:46.945280 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:46.945249    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:18:46.995971 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:46.995902    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b67c8b6c9-9crbf"]
Jul 28 17:18:48.261109 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:48.261074    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:53.099393 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:53.099345    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs\") pod \"network-metrics-daemon-zqjzm\" (UID: \"0ee703af-bd7c-45e1-8b8a-1b31645d40bf\") " pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:18:53.101664 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:53.101642    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee703af-bd7c-45e1-8b8a-1b31645d40bf-metrics-certs\") pod \"network-metrics-daemon-zqjzm\" (UID: \"0ee703af-bd7c-45e1-8b8a-1b31645d40bf\") " pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:18:53.323587 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:53.323549    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2k2q\""
Jul 28 17:18:53.331409 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:53.331381    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqjzm"
Jul 28 17:18:53.530110 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:53.530086    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zqjzm"]
Jul 28 17:18:53.532663 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:18:53.532638    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee703af_bd7c_45e1_8b8a_1b31645d40bf.slice/crio-59d56ef78aeb8331d59a940ef3faa7bbd9673edc972bbe1a75be8e73a0ef6434 WatchSource:0}: Error finding container 59d56ef78aeb8331d59a940ef3faa7bbd9673edc972bbe1a75be8e73a0ef6434: Status 404 returned error can't find the container with id 59d56ef78aeb8331d59a940ef3faa7bbd9673edc972bbe1a75be8e73a0ef6434
Jul 28 17:18:53.969211 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:53.969179    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zqjzm" event={"ID":"0ee703af-bd7c-45e1-8b8a-1b31645d40bf","Type":"ContainerStarted","Data":"59d56ef78aeb8331d59a940ef3faa7bbd9673edc972bbe1a75be8e73a0ef6434"}
Jul 28 17:18:54.976111 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:54.976069    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zqjzm" event={"ID":"0ee703af-bd7c-45e1-8b8a-1b31645d40bf","Type":"ContainerStarted","Data":"8a3e489f7aadf424fb6f061ae9a9ac620d1d33badcf5bfdb6bc012e87b6528de"}
Jul 28 17:18:55.985855 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:55.985819    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zqjzm" event={"ID":"0ee703af-bd7c-45e1-8b8a-1b31645d40bf","Type":"ContainerStarted","Data":"f3399ae6071b814d29e48d5f719fb397ed3d8d37b8dc6cce81bfe62a6d642040"}
Jul 28 17:18:56.003019 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:56.002971    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zqjzm" podStartSLOduration=129.719729687 podStartE2EDuration="2m11.002952418s" totalImagesPullingTime="1.283222731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:56.002710692 +0000 UTC m=+132.260320384" watchObservedRunningTime="2026-07-28 17:18:56.002952418 +0000 UTC m=+132.260562108"
Jul 28 17:18:58.999198 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:58.999107    2575 generic.go:356] "Generic (PLEG): container finished" podID="958ca081-580a-4d51-af0a-093e902e405e" containerID="75054e9d01991de34e20451b41f5d6affcc575009df4b92bc6dab1c61b033041" exitCode=0
Jul 28 17:18:58.999679 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:58.999185    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-vkqmg" event={"ID":"958ca081-580a-4d51-af0a-093e902e405e","Type":"ContainerDied","Data":"75054e9d01991de34e20451b41f5d6affcc575009df4b92bc6dab1c61b033041"}
Jul 28 17:18:58.999679 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:18:58.999643    2575 scope.go:117] "RemoveContainer" containerID="75054e9d01991de34e20451b41f5d6affcc575009df4b92bc6dab1c61b033041"
Jul 28 17:19:00.004758 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:00.004725    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-vkqmg" event={"ID":"958ca081-580a-4d51-af0a-093e902e405e","Type":"ContainerStarted","Data":"9c0759d589eb5674bf3eb4e6736827cfa424b4c5fcff1b2f417a3942a96d17a4"}
Jul 28 17:19:00.657234 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:00.657180    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-dfbf9859d-q5c99" podUID="a15ef051-198a-49eb-b21d-1d8a7b82ccba" containerName="registry" containerID="cri-o://315cbf9e7db50d9be04eaa735ce1eb5b24a6c9918f222216bf7ff855add1e78f" gracePeriod=30
Jul 28 17:19:00.955217 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:00.955188    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:19:01.010574 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.010543    2575 generic.go:356] "Generic (PLEG): container finished" podID="a15ef051-198a-49eb-b21d-1d8a7b82ccba" containerID="315cbf9e7db50d9be04eaa735ce1eb5b24a6c9918f222216bf7ff855add1e78f" exitCode=0
Jul 28 17:19:01.010574 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.010577    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dfbf9859d-q5c99"
Jul 28 17:19:01.010970 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.010587    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dfbf9859d-q5c99" event={"ID":"a15ef051-198a-49eb-b21d-1d8a7b82ccba","Type":"ContainerDied","Data":"315cbf9e7db50d9be04eaa735ce1eb5b24a6c9918f222216bf7ff855add1e78f"}
Jul 28 17:19:01.010970 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.010658    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dfbf9859d-q5c99" event={"ID":"a15ef051-198a-49eb-b21d-1d8a7b82ccba","Type":"ContainerDied","Data":"61862394a2bbcf0969dea990957d15ede4dca4c1d668bf46f9a45b2162d9b7f6"}
Jul 28 17:19:01.010970 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.010679    2575 scope.go:117] "RemoveContainer" containerID="315cbf9e7db50d9be04eaa735ce1eb5b24a6c9918f222216bf7ff855add1e78f"
Jul 28 17:19:01.020856 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.020834    2575 scope.go:117] "RemoveContainer" containerID="315cbf9e7db50d9be04eaa735ce1eb5b24a6c9918f222216bf7ff855add1e78f"
Jul 28 17:19:01.021129 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:19:01.021105    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"315cbf9e7db50d9be04eaa735ce1eb5b24a6c9918f222216bf7ff855add1e78f\": container with ID starting with 315cbf9e7db50d9be04eaa735ce1eb5b24a6c9918f222216bf7ff855add1e78f not found: ID does not exist" containerID="315cbf9e7db50d9be04eaa735ce1eb5b24a6c9918f222216bf7ff855add1e78f"
Jul 28 17:19:01.021230 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.021136    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315cbf9e7db50d9be04eaa735ce1eb5b24a6c9918f222216bf7ff855add1e78f"} err="failed to get container status \"315cbf9e7db50d9be04eaa735ce1eb5b24a6c9918f222216bf7ff855add1e78f\": rpc error: code = NotFound desc = could not find container \"315cbf9e7db50d9be04eaa735ce1eb5b24a6c9918f222216bf7ff855add1e78f\": container with ID starting with 315cbf9e7db50d9be04eaa735ce1eb5b24a6c9918f222216bf7ff855add1e78f not found: ID does not exist"
Jul 28 17:19:01.065407 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.065388    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a15ef051-198a-49eb-b21d-1d8a7b82ccba-image-registry-private-configuration\") pod \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") "
Jul 28 17:19:01.065501 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.065419    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a15ef051-198a-49eb-b21d-1d8a7b82ccba-trusted-ca\") pod \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") "
Jul 28 17:19:01.065501 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.065439    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a15ef051-198a-49eb-b21d-1d8a7b82ccba-installation-pull-secrets\") pod \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") "
Jul 28 17:19:01.065501 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.065473    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-certificates\") pod \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") "
Jul 28 17:19:01.065501 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.065497    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8bp9\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-kube-api-access-s8bp9\") pod \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") "
Jul 28 17:19:01.065707 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.065623    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls\") pod \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") "
Jul 28 17:19:01.065792 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.065765    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a15ef051-198a-49eb-b21d-1d8a7b82ccba-ca-trust-extracted\") pod \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") "
Jul 28 17:19:01.065862 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.065817    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-bound-sa-token\") pod \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\" (UID: \"a15ef051-198a-49eb-b21d-1d8a7b82ccba\") "
Jul 28 17:19:01.065920 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.065880    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15ef051-198a-49eb-b21d-1d8a7b82ccba-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a15ef051-198a-49eb-b21d-1d8a7b82ccba" (UID: "a15ef051-198a-49eb-b21d-1d8a7b82ccba"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:01.065967 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.065935    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a15ef051-198a-49eb-b21d-1d8a7b82ccba" (UID: "a15ef051-198a-49eb-b21d-1d8a7b82ccba"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:01.066543 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.066214    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a15ef051-198a-49eb-b21d-1d8a7b82ccba-trusted-ca\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:19:01.066543 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.066241    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-certificates\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:19:01.068430 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.068397    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a15ef051-198a-49eb-b21d-1d8a7b82ccba-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a15ef051-198a-49eb-b21d-1d8a7b82ccba" (UID: "a15ef051-198a-49eb-b21d-1d8a7b82ccba"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:01.068542 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.068422    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a15ef051-198a-49eb-b21d-1d8a7b82ccba" (UID: "a15ef051-198a-49eb-b21d-1d8a7b82ccba"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:19:01.068542 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.068448    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a15ef051-198a-49eb-b21d-1d8a7b82ccba" (UID: "a15ef051-198a-49eb-b21d-1d8a7b82ccba"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:19:01.068542 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.068482    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a15ef051-198a-49eb-b21d-1d8a7b82ccba-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a15ef051-198a-49eb-b21d-1d8a7b82ccba" (UID: "a15ef051-198a-49eb-b21d-1d8a7b82ccba"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:01.068679 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.068654    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-kube-api-access-s8bp9" (OuterVolumeSpecName: "kube-api-access-s8bp9") pod "a15ef051-198a-49eb-b21d-1d8a7b82ccba" (UID: "a15ef051-198a-49eb-b21d-1d8a7b82ccba"). InnerVolumeSpecName "kube-api-access-s8bp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:19:01.074474 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.074453    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15ef051-198a-49eb-b21d-1d8a7b82ccba-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a15ef051-198a-49eb-b21d-1d8a7b82ccba" (UID: "a15ef051-198a-49eb-b21d-1d8a7b82ccba"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:19:01.167561 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.167530    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8bp9\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-kube-api-access-s8bp9\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:19:01.167561 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.167560    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-registry-tls\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:19:01.167745 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.167570    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a15ef051-198a-49eb-b21d-1d8a7b82ccba-ca-trust-extracted\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:19:01.167745 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.167580    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a15ef051-198a-49eb-b21d-1d8a7b82ccba-bound-sa-token\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:19:01.167745 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.167591    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a15ef051-198a-49eb-b21d-1d8a7b82ccba-image-registry-private-configuration\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:19:01.167745 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.167600    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a15ef051-198a-49eb-b21d-1d8a7b82ccba-installation-pull-secrets\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:19:01.332301 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.332267    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dfbf9859d-q5c99"]
Jul 28 17:19:01.340464 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:01.340441    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-dfbf9859d-q5c99"]
Jul 28 17:19:02.310948 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:02.310909    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a15ef051-198a-49eb-b21d-1d8a7b82ccba" path="/var/lib/kubelet/pods/a15ef051-198a-49eb-b21d-1d8a7b82ccba/volumes"
Jul 28 17:19:12.014847 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.014785    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b67c8b6c9-9crbf" podUID="80ec7720-b034-4a97-9cc1-184f7b8ba541" containerName="console" containerID="cri-o://0a5f4e3cc63b156f840cedf9cf763a99221cca879e47b26af46e337952a36b18" gracePeriod=15
Jul 28 17:19:12.280993 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.280563    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:19:12.368981 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.368951    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjxj7\" (UniqueName: \"kubernetes.io/projected/80ec7720-b034-4a97-9cc1-184f7b8ba541-kube-api-access-cjxj7\") pod \"80ec7720-b034-4a97-9cc1-184f7b8ba541\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") "
Jul 28 17:19:12.369178 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.368992    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-service-ca\") pod \"80ec7720-b034-4a97-9cc1-184f7b8ba541\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") "
Jul 28 17:19:12.369178 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.369136    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-oauth-serving-cert\") pod \"80ec7720-b034-4a97-9cc1-184f7b8ba541\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") "
Jul 28 17:19:12.369178 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.369163    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-config\") pod \"80ec7720-b034-4a97-9cc1-184f7b8ba541\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") "
Jul 28 17:19:12.369330 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.369215    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-oauth-config\") pod \"80ec7720-b034-4a97-9cc1-184f7b8ba541\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") "
Jul 28 17:19:12.369330 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.369246    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-serving-cert\") pod \"80ec7720-b034-4a97-9cc1-184f7b8ba541\" (UID: \"80ec7720-b034-4a97-9cc1-184f7b8ba541\") "
Jul 28 17:19:12.369467 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.369437    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-service-ca" (OuterVolumeSpecName: "service-ca") pod "80ec7720-b034-4a97-9cc1-184f7b8ba541" (UID: "80ec7720-b034-4a97-9cc1-184f7b8ba541"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:12.369540 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.369443    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "80ec7720-b034-4a97-9cc1-184f7b8ba541" (UID: "80ec7720-b034-4a97-9cc1-184f7b8ba541"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:12.369540 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.369493    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-config" (OuterVolumeSpecName: "console-config") pod "80ec7720-b034-4a97-9cc1-184f7b8ba541" (UID: "80ec7720-b034-4a97-9cc1-184f7b8ba541"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:12.371808 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.371779    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ec7720-b034-4a97-9cc1-184f7b8ba541-kube-api-access-cjxj7" (OuterVolumeSpecName: "kube-api-access-cjxj7") pod "80ec7720-b034-4a97-9cc1-184f7b8ba541" (UID: "80ec7720-b034-4a97-9cc1-184f7b8ba541"). InnerVolumeSpecName "kube-api-access-cjxj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:19:12.371994 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.371966    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "80ec7720-b034-4a97-9cc1-184f7b8ba541" (UID: "80ec7720-b034-4a97-9cc1-184f7b8ba541"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:12.372051 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.371979    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "80ec7720-b034-4a97-9cc1-184f7b8ba541" (UID: "80ec7720-b034-4a97-9cc1-184f7b8ba541"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:12.469997 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.469960    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-oauth-serving-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:19:12.469997 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.469993    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:19:12.469997 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.470003    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-oauth-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:19:12.470298 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.470011    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80ec7720-b034-4a97-9cc1-184f7b8ba541-console-serving-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:19:12.470298 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.470020    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjxj7\" (UniqueName: \"kubernetes.io/projected/80ec7720-b034-4a97-9cc1-184f7b8ba541-kube-api-access-cjxj7\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:19:12.470298 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:12.470028    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80ec7720-b034-4a97-9cc1-184f7b8ba541-service-ca\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:19:13.063552 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:13.060296    2575 generic.go:356] "Generic (PLEG): container finished" podID="80ec7720-b034-4a97-9cc1-184f7b8ba541" containerID="0a5f4e3cc63b156f840cedf9cf763a99221cca879e47b26af46e337952a36b18" exitCode=0
Jul 28 17:19:13.063552 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:13.060398    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b67c8b6c9-9crbf" event={"ID":"80ec7720-b034-4a97-9cc1-184f7b8ba541","Type":"ContainerDied","Data":"0a5f4e3cc63b156f840cedf9cf763a99221cca879e47b26af46e337952a36b18"}
Jul 28 17:19:13.063552 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:13.060453    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b67c8b6c9-9crbf" event={"ID":"80ec7720-b034-4a97-9cc1-184f7b8ba541","Type":"ContainerDied","Data":"9d0b44b682cb79b6ae5aaa6ebf129075252cacbe5f3c54bed60855a431b6e762"}
Jul 28 17:19:13.063552 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:13.060485    2575 scope.go:117] "RemoveContainer" containerID="0a5f4e3cc63b156f840cedf9cf763a99221cca879e47b26af46e337952a36b18"
Jul 28 17:19:13.063552 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:13.060705    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b67c8b6c9-9crbf"
Jul 28 17:19:13.083956 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:13.083934    2575 scope.go:117] "RemoveContainer" containerID="0a5f4e3cc63b156f840cedf9cf763a99221cca879e47b26af46e337952a36b18"
Jul 28 17:19:13.084206 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:19:13.084186    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a5f4e3cc63b156f840cedf9cf763a99221cca879e47b26af46e337952a36b18\": container with ID starting with 0a5f4e3cc63b156f840cedf9cf763a99221cca879e47b26af46e337952a36b18 not found: ID does not exist" containerID="0a5f4e3cc63b156f840cedf9cf763a99221cca879e47b26af46e337952a36b18"
Jul 28 17:19:13.084269 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:13.084215    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a5f4e3cc63b156f840cedf9cf763a99221cca879e47b26af46e337952a36b18"} err="failed to get container status \"0a5f4e3cc63b156f840cedf9cf763a99221cca879e47b26af46e337952a36b18\": rpc error: code = NotFound desc = could not find container \"0a5f4e3cc63b156f840cedf9cf763a99221cca879e47b26af46e337952a36b18\": container with ID starting with 0a5f4e3cc63b156f840cedf9cf763a99221cca879e47b26af46e337952a36b18 not found: ID does not exist"
Jul 28 17:19:13.091151 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:13.091126    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b67c8b6c9-9crbf"]
Jul 28 17:19:13.094874 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:13.094851    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b67c8b6c9-9crbf"]
Jul 28 17:19:14.311166 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:14.311134    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80ec7720-b034-4a97-9cc1-184f7b8ba541" path="/var/lib/kubelet/pods/80ec7720-b034-4a97-9cc1-184f7b8ba541/volumes"
Jul 28 17:19:38.261809 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:38.261722    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.280889 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:38.280862    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:39.178309 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:19:39.178282    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:20:19.237879 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:19.237833    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dbff64df6-8bftp"]
Jul 28 17:20:44.262113 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.262063    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5dbff64df6-8bftp" podUID="981b07f5-ecb6-4d96-a63d-9575da136d8a" containerName="console" containerID="cri-o://f6fcceb8b83da5bc3e7025b7025d03cbe1356d382f505c1d819feee6db0c82a0" gracePeriod=15
Jul 28 17:20:44.408635 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.408603    2575 generic.go:356] "Generic (PLEG): container finished" podID="981b07f5-ecb6-4d96-a63d-9575da136d8a" containerID="f6fcceb8b83da5bc3e7025b7025d03cbe1356d382f505c1d819feee6db0c82a0" exitCode=0
Jul 28 17:20:44.408783 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.408691    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dbff64df6-8bftp" event={"ID":"981b07f5-ecb6-4d96-a63d-9575da136d8a","Type":"ContainerDied","Data":"f6fcceb8b83da5bc3e7025b7025d03cbe1356d382f505c1d819feee6db0c82a0"}
Jul 28 17:20:44.506729 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.506694    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:20:44.608870 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.608785    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-oauth-serving-cert\") pod \"981b07f5-ecb6-4d96-a63d-9575da136d8a\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") "
Jul 28 17:20:44.608870 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.608824    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-config\") pod \"981b07f5-ecb6-4d96-a63d-9575da136d8a\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") "
Jul 28 17:20:44.608870 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.608851    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hfs9\" (UniqueName: \"kubernetes.io/projected/981b07f5-ecb6-4d96-a63d-9575da136d8a-kube-api-access-5hfs9\") pod \"981b07f5-ecb6-4d96-a63d-9575da136d8a\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") "
Jul 28 17:20:44.609131 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.608882    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-service-ca\") pod \"981b07f5-ecb6-4d96-a63d-9575da136d8a\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") "
Jul 28 17:20:44.609131 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.608934    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-oauth-config\") pod \"981b07f5-ecb6-4d96-a63d-9575da136d8a\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") "
Jul 28 17:20:44.609131 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.608969    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-serving-cert\") pod \"981b07f5-ecb6-4d96-a63d-9575da136d8a\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") "
Jul 28 17:20:44.609285 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.609135    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-trusted-ca-bundle\") pod \"981b07f5-ecb6-4d96-a63d-9575da136d8a\" (UID: \"981b07f5-ecb6-4d96-a63d-9575da136d8a\") "
Jul 28 17:20:44.609340 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.609186    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "981b07f5-ecb6-4d96-a63d-9575da136d8a" (UID: "981b07f5-ecb6-4d96-a63d-9575da136d8a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:20:44.609340 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.609316    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-config" (OuterVolumeSpecName: "console-config") pod "981b07f5-ecb6-4d96-a63d-9575da136d8a" (UID: "981b07f5-ecb6-4d96-a63d-9575da136d8a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:20:44.609441 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.609360    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-service-ca" (OuterVolumeSpecName: "service-ca") pod "981b07f5-ecb6-4d96-a63d-9575da136d8a" (UID: "981b07f5-ecb6-4d96-a63d-9575da136d8a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:20:44.609441 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.609399    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "981b07f5-ecb6-4d96-a63d-9575da136d8a" (UID: "981b07f5-ecb6-4d96-a63d-9575da136d8a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:20:44.609557 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.609448    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-oauth-serving-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:20:44.609557 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.609465    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:20:44.609557 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.609479    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-service-ca\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:20:44.610951 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.610921    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981b07f5-ecb6-4d96-a63d-9575da136d8a-kube-api-access-5hfs9" (OuterVolumeSpecName: "kube-api-access-5hfs9") pod "981b07f5-ecb6-4d96-a63d-9575da136d8a" (UID: "981b07f5-ecb6-4d96-a63d-9575da136d8a"). InnerVolumeSpecName "kube-api-access-5hfs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:20:44.611063 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.611015    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "981b07f5-ecb6-4d96-a63d-9575da136d8a" (UID: "981b07f5-ecb6-4d96-a63d-9575da136d8a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:20:44.611183 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.611158    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "981b07f5-ecb6-4d96-a63d-9575da136d8a" (UID: "981b07f5-ecb6-4d96-a63d-9575da136d8a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:20:44.710011 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.709974    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hfs9\" (UniqueName: \"kubernetes.io/projected/981b07f5-ecb6-4d96-a63d-9575da136d8a-kube-api-access-5hfs9\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:20:44.710011 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.710002    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-oauth-config\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:20:44.710011 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.710011    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/981b07f5-ecb6-4d96-a63d-9575da136d8a-console-serving-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:20:44.710249 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:44.710022    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981b07f5-ecb6-4d96-a63d-9575da136d8a-trusted-ca-bundle\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:20:45.413384 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:45.413339    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dbff64df6-8bftp" event={"ID":"981b07f5-ecb6-4d96-a63d-9575da136d8a","Type":"ContainerDied","Data":"45ed638c1c2a8e412faa714921521b74f286522b984992abbf2bd12aa4dafa4d"}
Jul 28 17:20:45.413384 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:45.413368    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dbff64df6-8bftp"
Jul 28 17:20:45.413926 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:45.413411    2575 scope.go:117] "RemoveContainer" containerID="f6fcceb8b83da5bc3e7025b7025d03cbe1356d382f505c1d819feee6db0c82a0"
Jul 28 17:20:45.433409 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:45.433383    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dbff64df6-8bftp"]
Jul 28 17:20:45.435395 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:45.435371    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dbff64df6-8bftp"]
Jul 28 17:20:46.311419 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:20:46.311386    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="981b07f5-ecb6-4d96-a63d-9575da136d8a" path="/var/lib/kubelet/pods/981b07f5-ecb6-4d96-a63d-9575da136d8a/volumes"
Jul 28 17:21:44.155787 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:21:44.155751    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:21:44.162830 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:21:44.162804    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:21:44.169008 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:21:44.168952    2575 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:23:02.258532 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.258482    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-msmzj"]
Jul 28 17:23:02.258995 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.258789    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a15ef051-198a-49eb-b21d-1d8a7b82ccba" containerName="registry"
Jul 28 17:23:02.258995 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.258802    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a15ef051-198a-49eb-b21d-1d8a7b82ccba" containerName="registry"
Jul 28 17:23:02.258995 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.258813    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ec7720-b034-4a97-9cc1-184f7b8ba541" containerName="console"
Jul 28 17:23:02.258995 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.258818    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ec7720-b034-4a97-9cc1-184f7b8ba541" containerName="console"
Jul 28 17:23:02.258995 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.258835    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981b07f5-ecb6-4d96-a63d-9575da136d8a" containerName="console"
Jul 28 17:23:02.258995 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.258841    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="981b07f5-ecb6-4d96-a63d-9575da136d8a" containerName="console"
Jul 28 17:23:02.258995 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.258886    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a15ef051-198a-49eb-b21d-1d8a7b82ccba" containerName="registry"
Jul 28 17:23:02.258995 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.258918    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="981b07f5-ecb6-4d96-a63d-9575da136d8a" containerName="console"
Jul 28 17:23:02.258995 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.258925    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ec7720-b034-4a97-9cc1-184f7b8ba541" containerName="console"
Jul 28 17:23:02.260826 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.260809    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-msmzj"
Jul 28 17:23:02.263055 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.263029    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 17:23:02.263201 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.263166    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-zqwjr\""
Jul 28 17:23:02.263785 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.263766    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:23:02.263867 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.263771    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:23:02.272065 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.272044    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-msmzj"]
Jul 28 17:23:02.417794 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.417756    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjs47\" (UniqueName: \"kubernetes.io/projected/68c368f9-042f-4db6-99ad-22d0acd0721a-kube-api-access-mjs47\") pod \"seaweedfs-64568bcd49-msmzj\" (UID: \"68c368f9-042f-4db6-99ad-22d0acd0721a\") " pod="kserve/seaweedfs-64568bcd49-msmzj"
Jul 28 17:23:02.417956 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.417822    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/68c368f9-042f-4db6-99ad-22d0acd0721a-data\") pod \"seaweedfs-64568bcd49-msmzj\" (UID: \"68c368f9-042f-4db6-99ad-22d0acd0721a\") " pod="kserve/seaweedfs-64568bcd49-msmzj"
Jul 28 17:23:02.518308 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.518204    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/68c368f9-042f-4db6-99ad-22d0acd0721a-data\") pod \"seaweedfs-64568bcd49-msmzj\" (UID: \"68c368f9-042f-4db6-99ad-22d0acd0721a\") " pod="kserve/seaweedfs-64568bcd49-msmzj"
Jul 28 17:23:02.518470 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.518312    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjs47\" (UniqueName: \"kubernetes.io/projected/68c368f9-042f-4db6-99ad-22d0acd0721a-kube-api-access-mjs47\") pod \"seaweedfs-64568bcd49-msmzj\" (UID: \"68c368f9-042f-4db6-99ad-22d0acd0721a\") " pod="kserve/seaweedfs-64568bcd49-msmzj"
Jul 28 17:23:02.518674 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.518651    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/68c368f9-042f-4db6-99ad-22d0acd0721a-data\") pod \"seaweedfs-64568bcd49-msmzj\" (UID: \"68c368f9-042f-4db6-99ad-22d0acd0721a\") " pod="kserve/seaweedfs-64568bcd49-msmzj"
Jul 28 17:23:02.526565 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.526538    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjs47\" (UniqueName: \"kubernetes.io/projected/68c368f9-042f-4db6-99ad-22d0acd0721a-kube-api-access-mjs47\") pod \"seaweedfs-64568bcd49-msmzj\" (UID: \"68c368f9-042f-4db6-99ad-22d0acd0721a\") " pod="kserve/seaweedfs-64568bcd49-msmzj"
Jul 28 17:23:02.569716 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.569692    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-msmzj"
Jul 28 17:23:02.690021 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.689989    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-msmzj"]
Jul 28 17:23:02.692925 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:23:02.692896    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c368f9_042f_4db6_99ad_22d0acd0721a.slice/crio-1a54aa7b2a12fcc1a20b40a6831e6e55fa63b356dc81af83cf2a6eb37a00f05a WatchSource:0}: Error finding container 1a54aa7b2a12fcc1a20b40a6831e6e55fa63b356dc81af83cf2a6eb37a00f05a: Status 404 returned error can't find the container with id 1a54aa7b2a12fcc1a20b40a6831e6e55fa63b356dc81af83cf2a6eb37a00f05a
Jul 28 17:23:02.694570 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.694551    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:23:02.920329 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:02.920294    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-msmzj" event={"ID":"68c368f9-042f-4db6-99ad-22d0acd0721a","Type":"ContainerStarted","Data":"1a54aa7b2a12fcc1a20b40a6831e6e55fa63b356dc81af83cf2a6eb37a00f05a"}
Jul 28 17:23:05.934504 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:05.934474    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-msmzj" event={"ID":"68c368f9-042f-4db6-99ad-22d0acd0721a","Type":"ContainerStarted","Data":"041360816a02323366dd4cb2c6960f5dd90d23dd1bd35a235e347cd5928deea4"}
Jul 28 17:23:05.934886 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:05.934583    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-msmzj"
Jul 28 17:23:11.941552 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:11.941506    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-msmzj"
Jul 28 17:23:11.958300 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:23:11.958235    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-msmzj" podStartSLOduration=7.371851014 podStartE2EDuration="9.958216332s" totalImagesPullingTime="2.586365318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:05.949881026 +0000 UTC m=+382.207490716" watchObservedRunningTime="2026-07-28 17:23:11.958216332 +0000 UTC m=+388.215826025"
Jul 28 17:24:05.537921 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:05.537883    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-r9j77"]
Jul 28 17:24:05.540271 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:05.540254    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9j77"
Jul 28 17:24:05.542427 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:05.542397    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 17:24:05.542550 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:05.542406    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 17:24:05.542550 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:05.542500    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4q7m5\""
Jul 28 17:24:05.553344 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:05.553321    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-r9j77"]
Jul 28 17:24:05.619386 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:05.619361    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f9db20a-bc49-44c4-8c4c-e39d49a997be-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-r9j77\" (UID: \"3f9db20a-bc49-44c4-8c4c-e39d49a997be\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9j77"
Jul 28 17:24:05.619544 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:05.619461    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82xhr\" (UniqueName: \"kubernetes.io/projected/3f9db20a-bc49-44c4-8c4c-e39d49a997be-kube-api-access-82xhr\") pod \"cert-manager-cainjector-559cdd8c87-r9j77\" (UID: \"3f9db20a-bc49-44c4-8c4c-e39d49a997be\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9j77"
Jul 28 17:24:05.719932 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:05.719899    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82xhr\" (UniqueName: \"kubernetes.io/projected/3f9db20a-bc49-44c4-8c4c-e39d49a997be-kube-api-access-82xhr\") pod \"cert-manager-cainjector-559cdd8c87-r9j77\" (UID: \"3f9db20a-bc49-44c4-8c4c-e39d49a997be\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9j77"
Jul 28 17:24:05.720075 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:05.719946    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f9db20a-bc49-44c4-8c4c-e39d49a997be-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-r9j77\" (UID: \"3f9db20a-bc49-44c4-8c4c-e39d49a997be\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9j77"
Jul 28 17:24:05.727945 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:05.727915    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f9db20a-bc49-44c4-8c4c-e39d49a997be-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-r9j77\" (UID: \"3f9db20a-bc49-44c4-8c4c-e39d49a997be\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9j77"
Jul 28 17:24:05.728091 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:05.728070    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82xhr\" (UniqueName: \"kubernetes.io/projected/3f9db20a-bc49-44c4-8c4c-e39d49a997be-kube-api-access-82xhr\") pod \"cert-manager-cainjector-559cdd8c87-r9j77\" (UID: \"3f9db20a-bc49-44c4-8c4c-e39d49a997be\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9j77"
Jul 28 17:24:05.849704 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:05.849626    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9j77"
Jul 28 17:24:05.969185 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:05.969157    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-r9j77"]
Jul 28 17:24:05.971616 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:24:05.971587    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f9db20a_bc49_44c4_8c4c_e39d49a997be.slice/crio-5e722291366a91615578e3aeec7f80a248466fe2dab8eb8107435379f360d342 WatchSource:0}: Error finding container 5e722291366a91615578e3aeec7f80a248466fe2dab8eb8107435379f360d342: Status 404 returned error can't find the container with id 5e722291366a91615578e3aeec7f80a248466fe2dab8eb8107435379f360d342
Jul 28 17:24:06.157628 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:06.157553    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9j77" event={"ID":"3f9db20a-bc49-44c4-8c4c-e39d49a997be","Type":"ContainerStarted","Data":"5e722291366a91615578e3aeec7f80a248466fe2dab8eb8107435379f360d342"}
Jul 28 17:24:10.176087 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:10.176044    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9j77" event={"ID":"3f9db20a-bc49-44c4-8c4c-e39d49a997be","Type":"ContainerStarted","Data":"761d53c4d043f5c1b629ea0ecee6ef05b6656151598c2917020f447df0333013"}
Jul 28 17:24:10.192396 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:10.192346    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9j77" podStartSLOduration=1.848003371 podStartE2EDuration="5.192330261s" totalImagesPullingTime="3.34432689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:10.192301408 +0000 UTC m=+446.449911100" watchObservedRunningTime="2026-07-28 17:24:10.192330261 +0000 UTC m=+446.449939954"
Jul 28 17:24:11.413046 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:11.413011    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f69sv"]
Jul 28 17:24:11.415462 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:11.415444    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f69sv"
Jul 28 17:24:11.417626 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:11.417602    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fhdcd\""
Jul 28 17:24:11.426314 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:11.426290    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f69sv"]
Jul 28 17:24:11.572216 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:11.572175    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/652d6012-43ed-4c11-a591-1735158e48c0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f69sv\" (UID: \"652d6012-43ed-4c11-a591-1735158e48c0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f69sv"
Jul 28 17:24:11.572216 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:11.572215    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxndq\" (UniqueName: \"kubernetes.io/projected/652d6012-43ed-4c11-a591-1735158e48c0-kube-api-access-wxndq\") pod \"cert-manager-webhook-6486ffd997-f69sv\" (UID: \"652d6012-43ed-4c11-a591-1735158e48c0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f69sv"
Jul 28 17:24:11.672873 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:11.672787    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/652d6012-43ed-4c11-a591-1735158e48c0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f69sv\" (UID: \"652d6012-43ed-4c11-a591-1735158e48c0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f69sv"
Jul 28 17:24:11.672873 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:11.672827    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxndq\" (UniqueName: \"kubernetes.io/projected/652d6012-43ed-4c11-a591-1735158e48c0-kube-api-access-wxndq\") pod \"cert-manager-webhook-6486ffd997-f69sv\" (UID: \"652d6012-43ed-4c11-a591-1735158e48c0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f69sv"
Jul 28 17:24:11.681271 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:11.681240    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/652d6012-43ed-4c11-a591-1735158e48c0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f69sv\" (UID: \"652d6012-43ed-4c11-a591-1735158e48c0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f69sv"
Jul 28 17:24:11.681408 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:11.681304    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxndq\" (UniqueName: \"kubernetes.io/projected/652d6012-43ed-4c11-a591-1735158e48c0-kube-api-access-wxndq\") pod \"cert-manager-webhook-6486ffd997-f69sv\" (UID: \"652d6012-43ed-4c11-a591-1735158e48c0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f69sv"
Jul 28 17:24:11.724307 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:11.724274    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f69sv"
Jul 28 17:24:11.856375 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:11.856339    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f69sv"]
Jul 28 17:24:11.859217 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:24:11.859186    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod652d6012_43ed_4c11_a591_1735158e48c0.slice/crio-1383b7eb0af2e6c2849e176d6d024e6e8ffe617af9fd596e82cc0123060cb5f9 WatchSource:0}: Error finding container 1383b7eb0af2e6c2849e176d6d024e6e8ffe617af9fd596e82cc0123060cb5f9: Status 404 returned error can't find the container with id 1383b7eb0af2e6c2849e176d6d024e6e8ffe617af9fd596e82cc0123060cb5f9
Jul 28 17:24:12.186050 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:12.186013    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f69sv" event={"ID":"652d6012-43ed-4c11-a591-1735158e48c0","Type":"ContainerStarted","Data":"1c5acdbe9ddda073f3d9b62618e34ac512df398991573a905f184e84a8832211"}
Jul 28 17:24:12.186219 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:12.186061    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f69sv"
Jul 28 17:24:12.186219 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:12.186075    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f69sv" event={"ID":"652d6012-43ed-4c11-a591-1735158e48c0","Type":"ContainerStarted","Data":"1383b7eb0af2e6c2849e176d6d024e6e8ffe617af9fd596e82cc0123060cb5f9"}
Jul 28 17:24:12.203817 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:12.203758    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-f69sv" podStartSLOduration=1.2037378539999999 podStartE2EDuration="1.203737854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:12.202997763 +0000 UTC m=+448.460607453" watchObservedRunningTime="2026-07-28 17:24:12.203737854 +0000 UTC m=+448.461347545"
Jul 28 17:24:18.191891 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:18.191859    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f69sv"
Jul 28 17:24:21.159826 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:21.159790    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9978k"]
Jul 28 17:24:21.162299 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:21.162282    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9978k"
Jul 28 17:24:21.164514 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:21.164494    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 17:24:21.165309 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:21.165292    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:24:21.165409 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:21.165293    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xp5r5\""
Jul 28 17:24:21.172977 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:21.172953    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9978k"]
Jul 28 17:24:21.266116 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:21.266083    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c96dddf3-5664-4d4c-8ea5-564e8ce807b6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9978k\" (UID: \"c96dddf3-5664-4d4c-8ea5-564e8ce807b6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9978k"
Jul 28 17:24:21.266298 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:21.266168    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22t8f\" (UniqueName: \"kubernetes.io/projected/c96dddf3-5664-4d4c-8ea5-564e8ce807b6-kube-api-access-22t8f\") pod \"openshift-lws-operator-fd8ccff4c-9978k\" (UID: \"c96dddf3-5664-4d4c-8ea5-564e8ce807b6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9978k"
Jul 28 17:24:21.366589 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:21.366561    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22t8f\" (UniqueName: \"kubernetes.io/projected/c96dddf3-5664-4d4c-8ea5-564e8ce807b6-kube-api-access-22t8f\") pod \"openshift-lws-operator-fd8ccff4c-9978k\" (UID: \"c96dddf3-5664-4d4c-8ea5-564e8ce807b6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9978k"
Jul 28 17:24:21.366748 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:21.366608    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c96dddf3-5664-4d4c-8ea5-564e8ce807b6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9978k\" (UID: \"c96dddf3-5664-4d4c-8ea5-564e8ce807b6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9978k"
Jul 28 17:24:21.366950 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:21.366934    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c96dddf3-5664-4d4c-8ea5-564e8ce807b6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9978k\" (UID: \"c96dddf3-5664-4d4c-8ea5-564e8ce807b6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9978k"
Jul 28 17:24:21.375012 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:21.374979    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22t8f\" (UniqueName: \"kubernetes.io/projected/c96dddf3-5664-4d4c-8ea5-564e8ce807b6-kube-api-access-22t8f\") pod \"openshift-lws-operator-fd8ccff4c-9978k\" (UID: \"c96dddf3-5664-4d4c-8ea5-564e8ce807b6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9978k"
Jul 28 17:24:21.472038 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:21.472004    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9978k"
Jul 28 17:24:21.601265 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:21.601232    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9978k"]
Jul 28 17:24:21.605569 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:24:21.605505    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc96dddf3_5664_4d4c_8ea5_564e8ce807b6.slice/crio-d92e95a096a66afe995e4aece502fb9ff025299173ed07863e20a87924c2b54b WatchSource:0}: Error finding container d92e95a096a66afe995e4aece502fb9ff025299173ed07863e20a87924c2b54b: Status 404 returned error can't find the container with id d92e95a096a66afe995e4aece502fb9ff025299173ed07863e20a87924c2b54b
Jul 28 17:24:22.224463 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:22.224429    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9978k" event={"ID":"c96dddf3-5664-4d4c-8ea5-564e8ce807b6","Type":"ContainerStarted","Data":"d92e95a096a66afe995e4aece502fb9ff025299173ed07863e20a87924c2b54b"}
Jul 28 17:24:24.234939 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:24.234901    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9978k" event={"ID":"c96dddf3-5664-4d4c-8ea5-564e8ce807b6","Type":"ContainerStarted","Data":"b05c189693a9f684a7b574a4a68214f941b730ba37b606a3f929f691cbf83261"}
Jul 28 17:24:24.252159 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:24.252106    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9978k" podStartSLOduration=0.779887148 podStartE2EDuration="3.252091614s" totalImagesPullingTime="2.472204466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:24.250692327 +0000 UTC m=+460.508302019" watchObservedRunningTime="2026-07-28 17:24:24.252091614 +0000 UTC m=+460.509701304"
Jul 28 17:24:53.740509 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.740473    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"]
Jul 28 17:24:53.744711 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.744690    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:24:53.747869 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.747838    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 17:24:53.747869 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.747864    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pxkbp\""
Jul 28 17:24:53.748038 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.747863    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 17:24:53.748038 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.747924    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 17:24:53.753699 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.753679    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"]
Jul 28 17:24:53.847238 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.847199    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a13ad18-f3e8-42f8-8294-a5521deee97a-manager-config\") pod \"lws-controller-manager-8994644bb-j4rkm\" (UID: \"8a13ad18-f3e8-42f8-8294-a5521deee97a\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:24:53.847444 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.847266    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a13ad18-f3e8-42f8-8294-a5521deee97a-cert\") pod \"lws-controller-manager-8994644bb-j4rkm\" (UID: \"8a13ad18-f3e8-42f8-8294-a5521deee97a\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:24:53.847444 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.847292    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a13ad18-f3e8-42f8-8294-a5521deee97a-metrics-cert\") pod \"lws-controller-manager-8994644bb-j4rkm\" (UID: \"8a13ad18-f3e8-42f8-8294-a5521deee97a\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:24:53.847444 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.847347    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cr9d\" (UniqueName: \"kubernetes.io/projected/8a13ad18-f3e8-42f8-8294-a5521deee97a-kube-api-access-5cr9d\") pod \"lws-controller-manager-8994644bb-j4rkm\" (UID: \"8a13ad18-f3e8-42f8-8294-a5521deee97a\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:24:53.948274 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.948238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a13ad18-f3e8-42f8-8294-a5521deee97a-cert\") pod \"lws-controller-manager-8994644bb-j4rkm\" (UID: \"8a13ad18-f3e8-42f8-8294-a5521deee97a\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:24:53.948274 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.948275    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a13ad18-f3e8-42f8-8294-a5521deee97a-metrics-cert\") pod \"lws-controller-manager-8994644bb-j4rkm\" (UID: \"8a13ad18-f3e8-42f8-8294-a5521deee97a\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:24:53.948489 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.948403    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cr9d\" (UniqueName: \"kubernetes.io/projected/8a13ad18-f3e8-42f8-8294-a5521deee97a-kube-api-access-5cr9d\") pod \"lws-controller-manager-8994644bb-j4rkm\" (UID: \"8a13ad18-f3e8-42f8-8294-a5521deee97a\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:24:53.948489 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.948479    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a13ad18-f3e8-42f8-8294-a5521deee97a-manager-config\") pod \"lws-controller-manager-8994644bb-j4rkm\" (UID: \"8a13ad18-f3e8-42f8-8294-a5521deee97a\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:24:53.949118 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.949095    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a13ad18-f3e8-42f8-8294-a5521deee97a-manager-config\") pod \"lws-controller-manager-8994644bb-j4rkm\" (UID: \"8a13ad18-f3e8-42f8-8294-a5521deee97a\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:24:53.950906 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.950884    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a13ad18-f3e8-42f8-8294-a5521deee97a-cert\") pod \"lws-controller-manager-8994644bb-j4rkm\" (UID: \"8a13ad18-f3e8-42f8-8294-a5521deee97a\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:24:53.950906 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.950894    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a13ad18-f3e8-42f8-8294-a5521deee97a-metrics-cert\") pod \"lws-controller-manager-8994644bb-j4rkm\" (UID: \"8a13ad18-f3e8-42f8-8294-a5521deee97a\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:24:53.962187 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:53.962160    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cr9d\" (UniqueName: \"kubernetes.io/projected/8a13ad18-f3e8-42f8-8294-a5521deee97a-kube-api-access-5cr9d\") pod \"lws-controller-manager-8994644bb-j4rkm\" (UID: \"8a13ad18-f3e8-42f8-8294-a5521deee97a\") " pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:24:54.054412 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:54.054322    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:24:54.187351 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:54.187323    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"]
Jul 28 17:24:54.189926 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:24:54.189893    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a13ad18_f3e8_42f8_8294_a5521deee97a.slice/crio-77f2712c33a4dc7734a97c379639b9a221700b867adfd2843d5ba6ad4d15ccbc WatchSource:0}: Error finding container 77f2712c33a4dc7734a97c379639b9a221700b867adfd2843d5ba6ad4d15ccbc: Status 404 returned error can't find the container with id 77f2712c33a4dc7734a97c379639b9a221700b867adfd2843d5ba6ad4d15ccbc
Jul 28 17:24:54.353224 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:54.353140    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm" event={"ID":"8a13ad18-f3e8-42f8-8294-a5521deee97a","Type":"ContainerStarted","Data":"77f2712c33a4dc7734a97c379639b9a221700b867adfd2843d5ba6ad4d15ccbc"}
Jul 28 17:24:56.363280 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:56.363245    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm" event={"ID":"8a13ad18-f3e8-42f8-8294-a5521deee97a","Type":"ContainerStarted","Data":"8545ae5419ae12e4cfd695b2326b5752ec84d718dddefbcb83ceb4a6a3aa0c20"}
Jul 28 17:24:56.363747 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:56.363382    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:24:56.381984 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:24:56.381929    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm" podStartSLOduration=1.8640066709999998 podStartE2EDuration="3.381912872s" totalImagesPullingTime="1.517906201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:56.380397465 +0000 UTC m=+492.638007156" watchObservedRunningTime="2026-07-28 17:24:56.381912872 +0000 UTC m=+492.639522565"
Jul 28 17:25:05.128333 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.128300    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c4fff4cb5-l85lf"]
Jul 28 17:25:05.131619 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.131594    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.134612 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.134591    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 17:25:05.135163 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.135144    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 17:25:05.135560 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.135502    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 17:25:05.135560 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.135556    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xc6lg\""
Jul 28 17:25:05.135680 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.135506    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 17:25:05.135944 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.135923    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 17:25:05.136090 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.136003    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 17:25:05.136090 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.136075    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 17:25:05.141097 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.141079    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 17:25:05.148404 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.148383    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c4fff4cb5-l85lf"]
Jul 28 17:25:05.247971 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.247933    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f93b4a6-8381-4aa9-83c1-995cd46ca111-console-config\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.248175 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.247980    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f93b4a6-8381-4aa9-83c1-995cd46ca111-console-serving-cert\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.248175 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.248064    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f93b4a6-8381-4aa9-83c1-995cd46ca111-trusted-ca-bundle\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.248175 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.248120    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f93b4a6-8381-4aa9-83c1-995cd46ca111-console-oauth-config\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.248175 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.248163    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f93b4a6-8381-4aa9-83c1-995cd46ca111-oauth-serving-cert\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.248405 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.248196    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjs79\" (UniqueName: \"kubernetes.io/projected/0f93b4a6-8381-4aa9-83c1-995cd46ca111-kube-api-access-sjs79\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.248405 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.248212    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f93b4a6-8381-4aa9-83c1-995cd46ca111-service-ca\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.349211 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.349166    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f93b4a6-8381-4aa9-83c1-995cd46ca111-trusted-ca-bundle\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.349407 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.349229    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f93b4a6-8381-4aa9-83c1-995cd46ca111-console-oauth-config\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.349407 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.349274    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f93b4a6-8381-4aa9-83c1-995cd46ca111-oauth-serving-cert\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.349407 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.349298    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjs79\" (UniqueName: \"kubernetes.io/projected/0f93b4a6-8381-4aa9-83c1-995cd46ca111-kube-api-access-sjs79\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.349407 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.349325    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f93b4a6-8381-4aa9-83c1-995cd46ca111-service-ca\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.349407 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.349365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f93b4a6-8381-4aa9-83c1-995cd46ca111-console-config\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.349407 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.349399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f93b4a6-8381-4aa9-83c1-995cd46ca111-console-serving-cert\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.353564 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.350911    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f93b4a6-8381-4aa9-83c1-995cd46ca111-trusted-ca-bundle\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.353564 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.351784    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f93b4a6-8381-4aa9-83c1-995cd46ca111-oauth-serving-cert\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.353564 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.352360    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f93b4a6-8381-4aa9-83c1-995cd46ca111-service-ca\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.355947 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.355916    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f93b4a6-8381-4aa9-83c1-995cd46ca111-console-serving-cert\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.356129 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.356073    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f93b4a6-8381-4aa9-83c1-995cd46ca111-console-config\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.356710 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.356685    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f93b4a6-8381-4aa9-83c1-995cd46ca111-console-oauth-config\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.376960 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.376929    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjs79\" (UniqueName: \"kubernetes.io/projected/0f93b4a6-8381-4aa9-83c1-995cd46ca111-kube-api-access-sjs79\") pod \"console-c4fff4cb5-l85lf\" (UID: \"0f93b4a6-8381-4aa9-83c1-995cd46ca111\") " pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.441436 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.441408    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:05.583252 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:05.583214    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c4fff4cb5-l85lf"]
Jul 28 17:25:05.586713 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:25:05.586667    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f93b4a6_8381_4aa9_83c1_995cd46ca111.slice/crio-10b1eb2e04eadf6fb7f74257570b465c2bb9cfe5827f54afaa1aabd6d0788c93 WatchSource:0}: Error finding container 10b1eb2e04eadf6fb7f74257570b465c2bb9cfe5827f54afaa1aabd6d0788c93: Status 404 returned error can't find the container with id 10b1eb2e04eadf6fb7f74257570b465c2bb9cfe5827f54afaa1aabd6d0788c93
Jul 28 17:25:06.403883 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:06.403844    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4fff4cb5-l85lf" event={"ID":"0f93b4a6-8381-4aa9-83c1-995cd46ca111","Type":"ContainerStarted","Data":"83c57eba5a5fb46d116c24639555197cdb8e02bb13c949a59a7c85e1c702b5be"}
Jul 28 17:25:06.404237 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:06.403902    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4fff4cb5-l85lf" event={"ID":"0f93b4a6-8381-4aa9-83c1-995cd46ca111","Type":"ContainerStarted","Data":"10b1eb2e04eadf6fb7f74257570b465c2bb9cfe5827f54afaa1aabd6d0788c93"}
Jul 28 17:25:06.431947 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:06.431898    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c4fff4cb5-l85lf" podStartSLOduration=1.431883087 podStartE2EDuration="1.431883087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:06.431305507 +0000 UTC m=+502.688915197" watchObservedRunningTime="2026-07-28 17:25:06.431883087 +0000 UTC m=+502.689492777"
Jul 28 17:25:07.369349 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:07.369311    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8994644bb-j4rkm"
Jul 28 17:25:09.519382 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:09.519348    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c"]
Jul 28 17:25:09.522716 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:09.522691    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c"
Jul 28 17:25:09.525661 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:09.525641    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 17:25:09.525786 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:09.525719    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 17:25:09.525786 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:09.525769    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2bkc2\""
Jul 28 17:25:09.542998 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:09.542965    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c"]
Jul 28 17:25:09.687369 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:09.687334    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sftpp\" (UniqueName: \"kubernetes.io/projected/f48667b0-bf40-4193-890e-9acda58c29f3-kube-api-access-sftpp\") pod \"kuadrant-operator-controller-manager-66866f5fcc-2xg2c\" (UID: \"f48667b0-bf40-4193-890e-9acda58c29f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c"
Jul 28 17:25:09.687514 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:09.687374    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f48667b0-bf40-4193-890e-9acda58c29f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-2xg2c\" (UID: \"f48667b0-bf40-4193-890e-9acda58c29f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c"
Jul 28 17:25:09.788565 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:09.788456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sftpp\" (UniqueName: \"kubernetes.io/projected/f48667b0-bf40-4193-890e-9acda58c29f3-kube-api-access-sftpp\") pod \"kuadrant-operator-controller-manager-66866f5fcc-2xg2c\" (UID: \"f48667b0-bf40-4193-890e-9acda58c29f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c"
Jul 28 17:25:09.788565 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:09.788498    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f48667b0-bf40-4193-890e-9acda58c29f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-2xg2c\" (UID: \"f48667b0-bf40-4193-890e-9acda58c29f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c"
Jul 28 17:25:09.788926 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:09.788907    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f48667b0-bf40-4193-890e-9acda58c29f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-2xg2c\" (UID: \"f48667b0-bf40-4193-890e-9acda58c29f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c"
Jul 28 17:25:09.799775 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:09.799747    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sftpp\" (UniqueName: \"kubernetes.io/projected/f48667b0-bf40-4193-890e-9acda58c29f3-kube-api-access-sftpp\") pod \"kuadrant-operator-controller-manager-66866f5fcc-2xg2c\" (UID: \"f48667b0-bf40-4193-890e-9acda58c29f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c"
Jul 28 17:25:09.833308 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:09.833283    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c"
Jul 28 17:25:09.979160 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:25:09.979129    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48667b0_bf40_4193_890e_9acda58c29f3.slice/crio-5d8f82f686f3766fe0a442d8b822d58c1cd63fbe1e90faae29815f6d63bb6e6d WatchSource:0}: Error finding container 5d8f82f686f3766fe0a442d8b822d58c1cd63fbe1e90faae29815f6d63bb6e6d: Status 404 returned error can't find the container with id 5d8f82f686f3766fe0a442d8b822d58c1cd63fbe1e90faae29815f6d63bb6e6d
Jul 28 17:25:09.983452 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:09.983424    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c"]
Jul 28 17:25:10.422038 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:10.422004    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c" event={"ID":"f48667b0-bf40-4193-890e-9acda58c29f3","Type":"ContainerStarted","Data":"5d8f82f686f3766fe0a442d8b822d58c1cd63fbe1e90faae29815f6d63bb6e6d"}
Jul 28 17:25:14.445805 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:14.445771    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c" event={"ID":"f48667b0-bf40-4193-890e-9acda58c29f3","Type":"ContainerStarted","Data":"66fced8063323e96a63660bdd759ed44e00bc1c7cc3e52833fe47bbb9f06822e"}
Jul 28 17:25:14.446309 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:14.445873    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c"
Jul 28 17:25:14.468271 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:14.468210    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c" podStartSLOduration=1.5277793750000002 podStartE2EDuration="5.468189978s" totalImagesPullingTime="3.940410603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:14.467055095 +0000 UTC m=+510.724664785" watchObservedRunningTime="2026-07-28 17:25:14.468189978 +0000 UTC m=+510.725799668"
Jul 28 17:25:15.441979 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:15.441929    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:15.442194 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:15.442018    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:15.447071 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:15.447046    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:15.454184 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:15.454164    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c4fff4cb5-l85lf"
Jul 28 17:25:25.452568 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:25:25.452533    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2xg2c"
Jul 28 17:26:00.668687 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:00.668651    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"]
Jul 28 17:26:00.673829 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:00.673811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"
Jul 28 17:26:00.676140 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:00.676122    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 17:26:00.676214 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:00.676123    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cbt59\""
Jul 28 17:26:00.681163 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:00.680800    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"]
Jul 28 17:26:00.762259 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:00.762227    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"]
Jul 28 17:26:00.831728 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:00.831695    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmv6s\" (UniqueName: \"kubernetes.io/projected/d0035fb2-37b1-4614-9137-6e5493622d82-kube-api-access-bmv6s\") pod \"limitador-limitador-859c4d96d6-zgfl4\" (UID: \"d0035fb2-37b1-4614-9137-6e5493622d82\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"
Jul 28 17:26:00.831898 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:00.831788    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d0035fb2-37b1-4614-9137-6e5493622d82-config-file\") pod \"limitador-limitador-859c4d96d6-zgfl4\" (UID: \"d0035fb2-37b1-4614-9137-6e5493622d82\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"
Jul 28 17:26:00.932942 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:00.932874    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmv6s\" (UniqueName: \"kubernetes.io/projected/d0035fb2-37b1-4614-9137-6e5493622d82-kube-api-access-bmv6s\") pod \"limitador-limitador-859c4d96d6-zgfl4\" (UID: \"d0035fb2-37b1-4614-9137-6e5493622d82\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"
Jul 28 17:26:00.933072 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:00.932943    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d0035fb2-37b1-4614-9137-6e5493622d82-config-file\") pod \"limitador-limitador-859c4d96d6-zgfl4\" (UID: \"d0035fb2-37b1-4614-9137-6e5493622d82\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"
Jul 28 17:26:00.933648 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:00.933626    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d0035fb2-37b1-4614-9137-6e5493622d82-config-file\") pod \"limitador-limitador-859c4d96d6-zgfl4\" (UID: \"d0035fb2-37b1-4614-9137-6e5493622d82\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"
Jul 28 17:26:00.943710 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:00.943691    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmv6s\" (UniqueName: \"kubernetes.io/projected/d0035fb2-37b1-4614-9137-6e5493622d82-kube-api-access-bmv6s\") pod \"limitador-limitador-859c4d96d6-zgfl4\" (UID: \"d0035fb2-37b1-4614-9137-6e5493622d82\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"
Jul 28 17:26:00.985243 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:00.985213    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"
Jul 28 17:26:01.117729 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:01.117659    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"]
Jul 28 17:26:01.120282 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:26:01.120255    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0035fb2_37b1_4614_9137_6e5493622d82.slice/crio-cb5ee01d32f295af9f62446c2ef9b5631f9f16745350c35ea9e1e8548139c489 WatchSource:0}: Error finding container cb5ee01d32f295af9f62446c2ef9b5631f9f16745350c35ea9e1e8548139c489: Status 404 returned error can't find the container with id cb5ee01d32f295af9f62446c2ef9b5631f9f16745350c35ea9e1e8548139c489
Jul 28 17:26:01.650451 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:01.650416    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4" event={"ID":"d0035fb2-37b1-4614-9137-6e5493622d82","Type":"ContainerStarted","Data":"cb5ee01d32f295af9f62446c2ef9b5631f9f16745350c35ea9e1e8548139c489"}
Jul 28 17:26:02.657436 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:02.657403    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4" event={"ID":"d0035fb2-37b1-4614-9137-6e5493622d82","Type":"ContainerStarted","Data":"51b26563b248e6b3866cc475b7719cb5828a7824b40a15422b7bc96cef4d5f7d"}
Jul 28 17:26:02.657806 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:02.657565    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"
Jul 28 17:26:02.675628 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:02.675543    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4" podStartSLOduration=1.372962673 podStartE2EDuration="2.675508802s" totalImagesPullingTime="1.302546129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:26:02.674874239 +0000 UTC m=+558.932483932" watchObservedRunningTime="2026-07-28 17:26:02.675508802 +0000 UTC m=+558.933118492"
Jul 28 17:26:13.663655 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:13.663626    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"
Jul 28 17:26:13.911555 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:13.910727    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"]
Jul 28 17:26:13.911555 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:13.911077    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4" podUID="d0035fb2-37b1-4614-9137-6e5493622d82" containerName="limitador" containerID="cri-o://51b26563b248e6b3866cc475b7719cb5828a7824b40a15422b7bc96cef4d5f7d" gracePeriod=30
Jul 28 17:26:14.451147 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.451120    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"
Jul 28 17:26:14.543054 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.542977    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d0035fb2-37b1-4614-9137-6e5493622d82-config-file\") pod \"d0035fb2-37b1-4614-9137-6e5493622d82\" (UID: \"d0035fb2-37b1-4614-9137-6e5493622d82\") "
Jul 28 17:26:14.543201 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.543084    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmv6s\" (UniqueName: \"kubernetes.io/projected/d0035fb2-37b1-4614-9137-6e5493622d82-kube-api-access-bmv6s\") pod \"d0035fb2-37b1-4614-9137-6e5493622d82\" (UID: \"d0035fb2-37b1-4614-9137-6e5493622d82\") "
Jul 28 17:26:14.543380 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.543357    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0035fb2-37b1-4614-9137-6e5493622d82-config-file" (OuterVolumeSpecName: "config-file") pod "d0035fb2-37b1-4614-9137-6e5493622d82" (UID: "d0035fb2-37b1-4614-9137-6e5493622d82"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:26:14.544969 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.544943    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0035fb2-37b1-4614-9137-6e5493622d82-kube-api-access-bmv6s" (OuterVolumeSpecName: "kube-api-access-bmv6s") pod "d0035fb2-37b1-4614-9137-6e5493622d82" (UID: "d0035fb2-37b1-4614-9137-6e5493622d82"). InnerVolumeSpecName "kube-api-access-bmv6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:26:14.644732 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.644699    2575 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d0035fb2-37b1-4614-9137-6e5493622d82-config-file\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:26:14.644732 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.644727    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmv6s\" (UniqueName: \"kubernetes.io/projected/d0035fb2-37b1-4614-9137-6e5493622d82-kube-api-access-bmv6s\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:26:14.718633 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.718596    2575 generic.go:356] "Generic (PLEG): container finished" podID="d0035fb2-37b1-4614-9137-6e5493622d82" containerID="51b26563b248e6b3866cc475b7719cb5828a7824b40a15422b7bc96cef4d5f7d" exitCode=0
Jul 28 17:26:14.719088 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.718643    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"
Jul 28 17:26:14.719088 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.718673    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4" event={"ID":"d0035fb2-37b1-4614-9137-6e5493622d82","Type":"ContainerDied","Data":"51b26563b248e6b3866cc475b7719cb5828a7824b40a15422b7bc96cef4d5f7d"}
Jul 28 17:26:14.719088 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.718724    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zgfl4" event={"ID":"d0035fb2-37b1-4614-9137-6e5493622d82","Type":"ContainerDied","Data":"cb5ee01d32f295af9f62446c2ef9b5631f9f16745350c35ea9e1e8548139c489"}
Jul 28 17:26:14.719088 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.718756    2575 scope.go:117] "RemoveContainer" containerID="51b26563b248e6b3866cc475b7719cb5828a7824b40a15422b7bc96cef4d5f7d"
Jul 28 17:26:14.727296 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.727280    2575 scope.go:117] "RemoveContainer" containerID="51b26563b248e6b3866cc475b7719cb5828a7824b40a15422b7bc96cef4d5f7d"
Jul 28 17:26:14.727587 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:26:14.727507    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51b26563b248e6b3866cc475b7719cb5828a7824b40a15422b7bc96cef4d5f7d\": container with ID starting with 51b26563b248e6b3866cc475b7719cb5828a7824b40a15422b7bc96cef4d5f7d not found: ID does not exist" containerID="51b26563b248e6b3866cc475b7719cb5828a7824b40a15422b7bc96cef4d5f7d"
Jul 28 17:26:14.727666 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.727596    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b26563b248e6b3866cc475b7719cb5828a7824b40a15422b7bc96cef4d5f7d"} err="failed to get container status \"51b26563b248e6b3866cc475b7719cb5828a7824b40a15422b7bc96cef4d5f7d\": rpc error: code = NotFound desc = could not find container \"51b26563b248e6b3866cc475b7719cb5828a7824b40a15422b7bc96cef4d5f7d\": container with ID starting with 51b26563b248e6b3866cc475b7719cb5828a7824b40a15422b7bc96cef4d5f7d not found: ID does not exist"
Jul 28 17:26:14.740922 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.740898    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"]
Jul 28 17:26:14.745808 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:14.745789    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zgfl4"]
Jul 28 17:26:16.312268 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:16.312232    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0035fb2-37b1-4614-9137-6e5493622d82" path="/var/lib/kubelet/pods/d0035fb2-37b1-4614-9137-6e5493622d82/volumes"
Jul 28 17:26:31.238223 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:31.238181    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["observability/jaeger-6bbd7d4c9d-n8f2l"]
Jul 28 17:26:31.238676 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:31.238553    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0035fb2-37b1-4614-9137-6e5493622d82" containerName="limitador"
Jul 28 17:26:31.238676 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:31.238567    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0035fb2-37b1-4614-9137-6e5493622d82" containerName="limitador"
Jul 28 17:26:31.238676 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:31.238625    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0035fb2-37b1-4614-9137-6e5493622d82" containerName="limitador"
Jul 28 17:26:31.242843 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:31.242824    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-n8f2l"
Jul 28 17:26:31.245477 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:31.245455    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"openshift-service-ca.crt\""
Jul 28 17:26:31.246278 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:31.246259    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"observability\"/\"jaeger-dockercfg-hwgsp\""
Jul 28 17:26:31.246371 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:31.246306    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"kube-root-ca.crt\""
Jul 28 17:26:31.252101 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:31.252081    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-n8f2l"]
Jul 28 17:26:31.391114 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:31.391079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfh5j\" (UniqueName: \"kubernetes.io/projected/81f2a6aa-e3ad-4586-bb2d-717131a8e50e-kube-api-access-kfh5j\") pod \"jaeger-6bbd7d4c9d-n8f2l\" (UID: \"81f2a6aa-e3ad-4586-bb2d-717131a8e50e\") " pod="observability/jaeger-6bbd7d4c9d-n8f2l"
Jul 28 17:26:31.491579 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:31.491462    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfh5j\" (UniqueName: \"kubernetes.io/projected/81f2a6aa-e3ad-4586-bb2d-717131a8e50e-kube-api-access-kfh5j\") pod \"jaeger-6bbd7d4c9d-n8f2l\" (UID: \"81f2a6aa-e3ad-4586-bb2d-717131a8e50e\") " pod="observability/jaeger-6bbd7d4c9d-n8f2l"
Jul 28 17:26:31.504436 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:31.504410    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfh5j\" (UniqueName: \"kubernetes.io/projected/81f2a6aa-e3ad-4586-bb2d-717131a8e50e-kube-api-access-kfh5j\") pod \"jaeger-6bbd7d4c9d-n8f2l\" (UID: \"81f2a6aa-e3ad-4586-bb2d-717131a8e50e\") " pod="observability/jaeger-6bbd7d4c9d-n8f2l"
Jul 28 17:26:31.551994 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:31.551967    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-n8f2l"
Jul 28 17:26:31.686116 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:31.686076    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-n8f2l"]
Jul 28 17:26:31.689115 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:26:31.689086    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81f2a6aa_e3ad_4586_bb2d_717131a8e50e.slice/crio-fcbff8a70e259c3be3899fac4cc5b8f25c8ea07da2bed796d7c199491d02b230 WatchSource:0}: Error finding container fcbff8a70e259c3be3899fac4cc5b8f25c8ea07da2bed796d7c199491d02b230: Status 404 returned error can't find the container with id fcbff8a70e259c3be3899fac4cc5b8f25c8ea07da2bed796d7c199491d02b230
Jul 28 17:26:31.792239 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:31.792167    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-n8f2l" event={"ID":"81f2a6aa-e3ad-4586-bb2d-717131a8e50e","Type":"ContainerStarted","Data":"fcbff8a70e259c3be3899fac4cc5b8f25c8ea07da2bed796d7c199491d02b230"}
Jul 28 17:26:33.804695 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:33.804663    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-n8f2l" event={"ID":"81f2a6aa-e3ad-4586-bb2d-717131a8e50e","Type":"ContainerStarted","Data":"da5638366a7fb63ce45101d7e9a724084eac36ac29fbf7a5d6055d757eb795ca"}
Jul 28 17:26:33.805048 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:33.804722    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="observability/jaeger-6bbd7d4c9d-n8f2l"
Jul 28 17:26:33.823447 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:33.823398    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="observability/jaeger-6bbd7d4c9d-n8f2l" podStartSLOduration=1.134855421 podStartE2EDuration="2.823380493s" totalImagesPullingTime="1.688525072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:26:33.82211367 +0000 UTC m=+590.079723339" watchObservedRunningTime="2026-07-28 17:26:33.823380493 +0000 UTC m=+590.080990183"
Jul 28 17:26:34.811181 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:34.811153    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="observability/jaeger-6bbd7d4c9d-n8f2l"
Jul 28 17:26:44.188596 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:44.188567    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:26:44.193629 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:26:44.193604    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:27:09.757940 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.757870    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-phr94"]
Jul 28 17:27:09.761246 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.761220    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-phr94"
Jul 28 17:27:09.764350 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.764333    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-h7b96\""
Jul 28 17:27:09.764937 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.764922    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 17:27:09.775401 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.775369    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-zxn8g"]
Jul 28 17:27:09.780041 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.779867    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-phr94"]
Jul 28 17:27:09.780041 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.779996    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-zxn8g"
Jul 28 17:27:09.783749 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.783725    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 17:27:09.784783 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.784761    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-6tgj2\""
Jul 28 17:27:09.790934 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.790914    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-zxn8g"]
Jul 28 17:27:09.823042 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.823010    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xd7z\" (UniqueName: \"kubernetes.io/projected/6df280f9-a317-4ff5-a263-06c154c30f4b-kube-api-access-5xd7z\") pod \"model-serving-api-7b954ffc95-phr94\" (UID: \"6df280f9-a317-4ff5-a263-06c154c30f4b\") " pod="kserve/model-serving-api-7b954ffc95-phr94"
Jul 28 17:27:09.823181 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.823071    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9516887-ab55-4bdd-8f42-4a4f5404a174-cert\") pod \"odh-model-controller-75d9cdbddf-zxn8g\" (UID: \"c9516887-ab55-4bdd-8f42-4a4f5404a174\") " pod="kserve/odh-model-controller-75d9cdbddf-zxn8g"
Jul 28 17:27:09.823181 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.823126    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdggl\" (UniqueName: \"kubernetes.io/projected/c9516887-ab55-4bdd-8f42-4a4f5404a174-kube-api-access-bdggl\") pod \"odh-model-controller-75d9cdbddf-zxn8g\" (UID: \"c9516887-ab55-4bdd-8f42-4a4f5404a174\") " pod="kserve/odh-model-controller-75d9cdbddf-zxn8g"
Jul 28 17:27:09.823181 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.823179    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6df280f9-a317-4ff5-a263-06c154c30f4b-tls-certs\") pod \"model-serving-api-7b954ffc95-phr94\" (UID: \"6df280f9-a317-4ff5-a263-06c154c30f4b\") " pod="kserve/model-serving-api-7b954ffc95-phr94"
Jul 28 17:27:09.923904 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.923865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xd7z\" (UniqueName: \"kubernetes.io/projected/6df280f9-a317-4ff5-a263-06c154c30f4b-kube-api-access-5xd7z\") pod \"model-serving-api-7b954ffc95-phr94\" (UID: \"6df280f9-a317-4ff5-a263-06c154c30f4b\") " pod="kserve/model-serving-api-7b954ffc95-phr94"
Jul 28 17:27:09.924131 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.923925    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9516887-ab55-4bdd-8f42-4a4f5404a174-cert\") pod \"odh-model-controller-75d9cdbddf-zxn8g\" (UID: \"c9516887-ab55-4bdd-8f42-4a4f5404a174\") " pod="kserve/odh-model-controller-75d9cdbddf-zxn8g"
Jul 28 17:27:09.924131 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.923953    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdggl\" (UniqueName: \"kubernetes.io/projected/c9516887-ab55-4bdd-8f42-4a4f5404a174-kube-api-access-bdggl\") pod \"odh-model-controller-75d9cdbddf-zxn8g\" (UID: \"c9516887-ab55-4bdd-8f42-4a4f5404a174\") " pod="kserve/odh-model-controller-75d9cdbddf-zxn8g"
Jul 28 17:27:09.924131 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.923987    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6df280f9-a317-4ff5-a263-06c154c30f4b-tls-certs\") pod \"model-serving-api-7b954ffc95-phr94\" (UID: \"6df280f9-a317-4ff5-a263-06c154c30f4b\") " pod="kserve/model-serving-api-7b954ffc95-phr94"
Jul 28 17:27:09.924131 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:27:09.924074    2575 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 28 17:27:09.924426 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:27:09.924149    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9516887-ab55-4bdd-8f42-4a4f5404a174-cert podName:c9516887-ab55-4bdd-8f42-4a4f5404a174 nodeName:}" failed. No retries permitted until 2026-07-28 17:27:10.42412641 +0000 UTC m=+626.681736081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c9516887-ab55-4bdd-8f42-4a4f5404a174-cert") pod "odh-model-controller-75d9cdbddf-zxn8g" (UID: "c9516887-ab55-4bdd-8f42-4a4f5404a174") : secret "odh-model-controller-webhook-cert" not found
Jul 28 17:27:09.926878 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.926854    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6df280f9-a317-4ff5-a263-06c154c30f4b-tls-certs\") pod \"model-serving-api-7b954ffc95-phr94\" (UID: \"6df280f9-a317-4ff5-a263-06c154c30f4b\") " pod="kserve/model-serving-api-7b954ffc95-phr94"
Jul 28 17:27:09.934652 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.934627    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdggl\" (UniqueName: \"kubernetes.io/projected/c9516887-ab55-4bdd-8f42-4a4f5404a174-kube-api-access-bdggl\") pod \"odh-model-controller-75d9cdbddf-zxn8g\" (UID: \"c9516887-ab55-4bdd-8f42-4a4f5404a174\") " pod="kserve/odh-model-controller-75d9cdbddf-zxn8g"
Jul 28 17:27:09.934927 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:09.934911    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xd7z\" (UniqueName: \"kubernetes.io/projected/6df280f9-a317-4ff5-a263-06c154c30f4b-kube-api-access-5xd7z\") pod \"model-serving-api-7b954ffc95-phr94\" (UID: \"6df280f9-a317-4ff5-a263-06c154c30f4b\") " pod="kserve/model-serving-api-7b954ffc95-phr94"
Jul 28 17:27:10.071322 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:10.071231    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-phr94"
Jul 28 17:27:10.208680 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:10.208654    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-phr94"]
Jul 28 17:27:10.210843 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:27:10.210816    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df280f9_a317_4ff5_a263_06c154c30f4b.slice/crio-6d91848f6cb44bf40abf22315e9a6006304efcc0098de00f03fa266b84d6e028 WatchSource:0}: Error finding container 6d91848f6cb44bf40abf22315e9a6006304efcc0098de00f03fa266b84d6e028: Status 404 returned error can't find the container with id 6d91848f6cb44bf40abf22315e9a6006304efcc0098de00f03fa266b84d6e028
Jul 28 17:27:10.429397 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:10.429368    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9516887-ab55-4bdd-8f42-4a4f5404a174-cert\") pod \"odh-model-controller-75d9cdbddf-zxn8g\" (UID: \"c9516887-ab55-4bdd-8f42-4a4f5404a174\") " pod="kserve/odh-model-controller-75d9cdbddf-zxn8g"
Jul 28 17:27:10.431791 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:10.431761    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9516887-ab55-4bdd-8f42-4a4f5404a174-cert\") pod \"odh-model-controller-75d9cdbddf-zxn8g\" (UID: \"c9516887-ab55-4bdd-8f42-4a4f5404a174\") " pod="kserve/odh-model-controller-75d9cdbddf-zxn8g"
Jul 28 17:27:10.694625 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:10.694512    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-zxn8g"
Jul 28 17:27:10.853927 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:10.853890    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-zxn8g"]
Jul 28 17:27:10.857354 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:27:10.857318    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9516887_ab55_4bdd_8f42_4a4f5404a174.slice/crio-23e6a1b17ab65ace4ee3fcade35a724ec4de366182119348d57809bd610dc36e WatchSource:0}: Error finding container 23e6a1b17ab65ace4ee3fcade35a724ec4de366182119348d57809bd610dc36e: Status 404 returned error can't find the container with id 23e6a1b17ab65ace4ee3fcade35a724ec4de366182119348d57809bd610dc36e
Jul 28 17:27:10.976710 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:10.976616    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-phr94" event={"ID":"6df280f9-a317-4ff5-a263-06c154c30f4b","Type":"ContainerStarted","Data":"6d91848f6cb44bf40abf22315e9a6006304efcc0098de00f03fa266b84d6e028"}
Jul 28 17:27:10.978197 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:10.978168    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-zxn8g" event={"ID":"c9516887-ab55-4bdd-8f42-4a4f5404a174","Type":"ContainerStarted","Data":"23e6a1b17ab65ace4ee3fcade35a724ec4de366182119348d57809bd610dc36e"}
Jul 28 17:27:12.991228 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:12.991186    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-phr94" event={"ID":"6df280f9-a317-4ff5-a263-06c154c30f4b","Type":"ContainerStarted","Data":"6af8201f2d5cc9fc6d8b6013fb663c1fada5d484c08419f3fc16575e6ffb47ba"}
Jul 28 17:27:12.991699 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:12.991544    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-phr94"
Jul 28 17:27:13.011247 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:13.011185    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-phr94" podStartSLOduration=1.823732477 podStartE2EDuration="4.011166714s" totalImagesPullingTime="2.187434237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:13.008931075 +0000 UTC m=+629.266540769" watchObservedRunningTime="2026-07-28 17:27:13.011166714 +0000 UTC m=+629.268776404"
Jul 28 17:27:15.001767 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:15.001733    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-zxn8g" event={"ID":"c9516887-ab55-4bdd-8f42-4a4f5404a174","Type":"ContainerStarted","Data":"c91da65dd4fbbe76623a03d5f2a735435b1d494abb546b11a08a40f8f67290f9"}
Jul 28 17:27:15.002134 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:15.001869    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-zxn8g"
Jul 28 17:27:15.019279 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:15.019232    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-zxn8g" podStartSLOduration=2.292600985 podStartE2EDuration="6.019220208s" totalImagesPullingTime="3.726619223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:15.018669418 +0000 UTC m=+631.276279109" watchObservedRunningTime="2026-07-28 17:27:15.019220208 +0000 UTC m=+631.276829897"
Jul 28 17:27:24.000732 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:24.000704    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-phr94"
Jul 28 17:27:26.008949 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:27:26.008924    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-zxn8g"
Jul 28 17:29:17.476670 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.476639    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"]
Jul 28 17:29:17.479991 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.479970    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"
Jul 28 17:29:17.482351 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.482334    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 28 17:29:17.483289 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.483269    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-5bgfn\""
Jul 28 17:29:17.483423 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.483404    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 28 17:29:17.483546 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.483297    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-tksl9\""
Jul 28 17:29:17.483546 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.483365    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 28 17:29:17.492441 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.492422    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"]
Jul 28 17:29:17.625485 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.625448    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8t2j\" (UniqueName: \"kubernetes.io/projected/4eee7567-9df4-47d1-b48b-3087161cf60e-kube-api-access-m8t2j\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh\" (UID: \"4eee7567-9df4-47d1-b48b-3087161cf60e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"
Jul 28 17:29:17.625667 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.625536    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eee7567-9df4-47d1-b48b-3087161cf60e-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh\" (UID: \"4eee7567-9df4-47d1-b48b-3087161cf60e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"
Jul 28 17:29:17.726108 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.726075    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eee7567-9df4-47d1-b48b-3087161cf60e-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh\" (UID: \"4eee7567-9df4-47d1-b48b-3087161cf60e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"
Jul 28 17:29:17.726265 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.726156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8t2j\" (UniqueName: \"kubernetes.io/projected/4eee7567-9df4-47d1-b48b-3087161cf60e-kube-api-access-m8t2j\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh\" (UID: \"4eee7567-9df4-47d1-b48b-3087161cf60e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"
Jul 28 17:29:17.728746 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.728687    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eee7567-9df4-47d1-b48b-3087161cf60e-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh\" (UID: \"4eee7567-9df4-47d1-b48b-3087161cf60e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"
Jul 28 17:29:17.733935 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.733913    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8t2j\" (UniqueName: \"kubernetes.io/projected/4eee7567-9df4-47d1-b48b-3087161cf60e-kube-api-access-m8t2j\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh\" (UID: \"4eee7567-9df4-47d1-b48b-3087161cf60e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"
Jul 28 17:29:17.790377 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.790343    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"
Jul 28 17:29:17.922808 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.922783    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"]
Jul 28 17:29:17.925597 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:29:17.925568    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eee7567_9df4_47d1_b48b_3087161cf60e.slice/crio-9096eb259a4442e08005c00b12e19522ac684e456a905198082d6ca3bf4dc117 WatchSource:0}: Error finding container 9096eb259a4442e08005c00b12e19522ac684e456a905198082d6ca3bf4dc117: Status 404 returned error can't find the container with id 9096eb259a4442e08005c00b12e19522ac684e456a905198082d6ca3bf4dc117
Jul 28 17:29:17.927574 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:17.927557    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:29:18.546192 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:18.546155    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh" event={"ID":"4eee7567-9df4-47d1-b48b-3087161cf60e","Type":"ContainerStarted","Data":"9096eb259a4442e08005c00b12e19522ac684e456a905198082d6ca3bf4dc117"}
Jul 28 17:29:20.558059 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:20.558016    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh" event={"ID":"4eee7567-9df4-47d1-b48b-3087161cf60e","Type":"ContainerStarted","Data":"b2d9b091341c125355f57299ca95dce219ae88ca1df99203df73c649af6f8e78"}
Jul 28 17:29:20.558472 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:20.558103    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"
Jul 28 17:29:20.578962 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:20.578873    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh" podStartSLOduration=1.176261199 podStartE2EDuration="3.578857555s" totalImagesPullingTime="2.402596356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:29:20.576864955 +0000 UTC m=+756.834474645" watchObservedRunningTime="2026-07-28 17:29:20.578857555 +0000 UTC m=+756.836467260"
Jul 28 17:29:51.564567 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:29:51.564515    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"
Jul 28 17:30:05.373451 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:05.373416    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"]
Jul 28 17:30:11.564959 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:11.564873    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh" podUID="4eee7567-9df4-47d1-b48b-3087161cf60e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:30:20.392321 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.392242    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh" podUID="4eee7567-9df4-47d1-b48b-3087161cf60e" containerName="main" containerID="cri-o://b2d9b091341c125355f57299ca95dce219ae88ca1df99203df73c649af6f8e78" gracePeriod=45
Jul 28 17:30:20.638218 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.638187    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"
Jul 28 17:30:20.690179 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.690148    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eee7567-9df4-47d1-b48b-3087161cf60e-tls-certs\") pod \"4eee7567-9df4-47d1-b48b-3087161cf60e\" (UID: \"4eee7567-9df4-47d1-b48b-3087161cf60e\") "
Jul 28 17:30:20.690343 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.690197    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8t2j\" (UniqueName: \"kubernetes.io/projected/4eee7567-9df4-47d1-b48b-3087161cf60e-kube-api-access-m8t2j\") pod \"4eee7567-9df4-47d1-b48b-3087161cf60e\" (UID: \"4eee7567-9df4-47d1-b48b-3087161cf60e\") "
Jul 28 17:30:20.692285 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.692258    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eee7567-9df4-47d1-b48b-3087161cf60e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4eee7567-9df4-47d1-b48b-3087161cf60e" (UID: "4eee7567-9df4-47d1-b48b-3087161cf60e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:30:20.692400 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.692345    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eee7567-9df4-47d1-b48b-3087161cf60e-kube-api-access-m8t2j" (OuterVolumeSpecName: "kube-api-access-m8t2j") pod "4eee7567-9df4-47d1-b48b-3087161cf60e" (UID: "4eee7567-9df4-47d1-b48b-3087161cf60e"). InnerVolumeSpecName "kube-api-access-m8t2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:30:20.791782 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.791746    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eee7567-9df4-47d1-b48b-3087161cf60e-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:30:20.791782 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.791777    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8t2j\" (UniqueName: \"kubernetes.io/projected/4eee7567-9df4-47d1-b48b-3087161cf60e-kube-api-access-m8t2j\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:30:20.826697 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.826666    2575 generic.go:356] "Generic (PLEG): container finished" podID="4eee7567-9df4-47d1-b48b-3087161cf60e" containerID="b2d9b091341c125355f57299ca95dce219ae88ca1df99203df73c649af6f8e78" exitCode=0
Jul 28 17:30:20.826851 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.826718    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"
Jul 28 17:30:20.826851 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.826747    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh" event={"ID":"4eee7567-9df4-47d1-b48b-3087161cf60e","Type":"ContainerDied","Data":"b2d9b091341c125355f57299ca95dce219ae88ca1df99203df73c649af6f8e78"}
Jul 28 17:30:20.826851 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.826801    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh" event={"ID":"4eee7567-9df4-47d1-b48b-3087161cf60e","Type":"ContainerDied","Data":"9096eb259a4442e08005c00b12e19522ac684e456a905198082d6ca3bf4dc117"}
Jul 28 17:30:20.826851 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.826824    2575 scope.go:117] "RemoveContainer" containerID="b2d9b091341c125355f57299ca95dce219ae88ca1df99203df73c649af6f8e78"
Jul 28 17:30:20.835223 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.835208    2575 scope.go:117] "RemoveContainer" containerID="b2d9b091341c125355f57299ca95dce219ae88ca1df99203df73c649af6f8e78"
Jul 28 17:30:20.835450 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:30:20.835432    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d9b091341c125355f57299ca95dce219ae88ca1df99203df73c649af6f8e78\": container with ID starting with b2d9b091341c125355f57299ca95dce219ae88ca1df99203df73c649af6f8e78 not found: ID does not exist" containerID="b2d9b091341c125355f57299ca95dce219ae88ca1df99203df73c649af6f8e78"
Jul 28 17:30:20.835498 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.835456    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d9b091341c125355f57299ca95dce219ae88ca1df99203df73c649af6f8e78"} err="failed to get container status \"b2d9b091341c125355f57299ca95dce219ae88ca1df99203df73c649af6f8e78\": rpc error: code = NotFound desc = could not find container \"b2d9b091341c125355f57299ca95dce219ae88ca1df99203df73c649af6f8e78\": container with ID starting with b2d9b091341c125355f57299ca95dce219ae88ca1df99203df73c649af6f8e78 not found: ID does not exist"
Jul 28 17:30:20.848407 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.848385    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"]
Jul 28 17:30:20.852192 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:20.852171    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-zjkmh"]
Jul 28 17:30:22.311775 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:22.311741    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eee7567-9df4-47d1-b48b-3087161cf60e" path="/var/lib/kubelet/pods/4eee7567-9df4-47d1-b48b-3087161cf60e/volumes"
Jul 28 17:30:28.558542 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.558495    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"]
Jul 28 17:30:28.558979 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.558832    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eee7567-9df4-47d1-b48b-3087161cf60e" containerName="main"
Jul 28 17:30:28.558979 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.558845    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eee7567-9df4-47d1-b48b-3087161cf60e" containerName="main"
Jul 28 17:30:28.558979 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.558895    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eee7567-9df4-47d1-b48b-3087161cf60e" containerName="main"
Jul 28 17:30:28.563228 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.563206    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"
Jul 28 17:30:28.566392 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.566368    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-sxrg7\""
Jul 28 17:30:28.566539 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.566419    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 28 17:30:28.566539 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.566448    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 28 17:30:28.566539 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.566369    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-vc5sh\""
Jul 28 17:30:28.566539 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.566464    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 28 17:30:28.572162 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.572142    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"]
Jul 28 17:30:28.657272 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.657243    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kg5s\" (UniqueName: \"kubernetes.io/projected/156801bb-2ae0-465d-a3b4-42b371fc8933-kube-api-access-9kg5s\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq\" (UID: \"156801bb-2ae0-465d-a3b4-42b371fc8933\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"
Jul 28 17:30:28.657430 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.657355    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156801bb-2ae0-465d-a3b4-42b371fc8933-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq\" (UID: \"156801bb-2ae0-465d-a3b4-42b371fc8933\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"
Jul 28 17:30:28.758124 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.758087    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156801bb-2ae0-465d-a3b4-42b371fc8933-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq\" (UID: \"156801bb-2ae0-465d-a3b4-42b371fc8933\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"
Jul 28 17:30:28.758280 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.758147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kg5s\" (UniqueName: \"kubernetes.io/projected/156801bb-2ae0-465d-a3b4-42b371fc8933-kube-api-access-9kg5s\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq\" (UID: \"156801bb-2ae0-465d-a3b4-42b371fc8933\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"
Jul 28 17:30:28.760928 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.760889    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156801bb-2ae0-465d-a3b4-42b371fc8933-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq\" (UID: \"156801bb-2ae0-465d-a3b4-42b371fc8933\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"
Jul 28 17:30:28.768206 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.768177    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kg5s\" (UniqueName: \"kubernetes.io/projected/156801bb-2ae0-465d-a3b4-42b371fc8933-kube-api-access-9kg5s\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq\" (UID: \"156801bb-2ae0-465d-a3b4-42b371fc8933\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"
Jul 28 17:30:28.873941 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:28.873867    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"
Jul 28 17:30:29.004446 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:29.004418    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"]
Jul 28 17:30:29.007119 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:30:29.007095    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156801bb_2ae0_465d_a3b4_42b371fc8933.slice/crio-edabe143c8161b06835517e81432c6c933f45f18b181b954ab92b815f363c6d8 WatchSource:0}: Error finding container edabe143c8161b06835517e81432c6c933f45f18b181b954ab92b815f363c6d8: Status 404 returned error can't find the container with id edabe143c8161b06835517e81432c6c933f45f18b181b954ab92b815f363c6d8
Jul 28 17:30:29.868773 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:29.868736    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq" event={"ID":"156801bb-2ae0-465d-a3b4-42b371fc8933","Type":"ContainerStarted","Data":"5df9950af8399c6d110924cbd69055e953d883a8359cf9d1ba6bc023d663f27e"}
Jul 28 17:30:29.869228 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:29.868793    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq" event={"ID":"156801bb-2ae0-465d-a3b4-42b371fc8933","Type":"ContainerStarted","Data":"edabe143c8161b06835517e81432c6c933f45f18b181b954ab92b815f363c6d8"}
Jul 28 17:30:29.869228 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:29.868867    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"
Jul 28 17:30:29.885558 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:30:29.885484    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq" podStartSLOduration=1.885468983 podStartE2EDuration="1.885468983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:30:29.884436377 +0000 UTC m=+826.142046067" watchObservedRunningTime="2026-07-28 17:30:29.885468983 +0000 UTC m=+826.143078673"
Jul 28 17:31:00.875772 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:00.875743    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"
Jul 28 17:31:07.411871 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:07.411833    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"]
Jul 28 17:31:10.875483 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:10.875447    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq" podUID="156801bb-2ae0-465d-a3b4-42b371fc8933" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:31:20.876333 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:20.876294    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq" podUID="156801bb-2ae0-465d-a3b4-42b371fc8933" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:31:22.431271 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:22.431227    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq" podUID="156801bb-2ae0-465d-a3b4-42b371fc8933" containerName="main" containerID="cri-o://5df9950af8399c6d110924cbd69055e953d883a8359cf9d1ba6bc023d663f27e" gracePeriod=45
Jul 28 17:31:22.678124 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:22.678093    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"
Jul 28 17:31:22.692576 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:22.692493    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156801bb-2ae0-465d-a3b4-42b371fc8933-tls-certs\") pod \"156801bb-2ae0-465d-a3b4-42b371fc8933\" (UID: \"156801bb-2ae0-465d-a3b4-42b371fc8933\") "
Jul 28 17:31:22.692687 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:22.692646    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kg5s\" (UniqueName: \"kubernetes.io/projected/156801bb-2ae0-465d-a3b4-42b371fc8933-kube-api-access-9kg5s\") pod \"156801bb-2ae0-465d-a3b4-42b371fc8933\" (UID: \"156801bb-2ae0-465d-a3b4-42b371fc8933\") "
Jul 28 17:31:22.694764 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:22.694725    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/156801bb-2ae0-465d-a3b4-42b371fc8933-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "156801bb-2ae0-465d-a3b4-42b371fc8933" (UID: "156801bb-2ae0-465d-a3b4-42b371fc8933"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:31:22.694883 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:22.694802    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156801bb-2ae0-465d-a3b4-42b371fc8933-kube-api-access-9kg5s" (OuterVolumeSpecName: "kube-api-access-9kg5s") pod "156801bb-2ae0-465d-a3b4-42b371fc8933" (UID: "156801bb-2ae0-465d-a3b4-42b371fc8933"). InnerVolumeSpecName "kube-api-access-9kg5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:31:22.793974 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:22.793940    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kg5s\" (UniqueName: \"kubernetes.io/projected/156801bb-2ae0-465d-a3b4-42b371fc8933-kube-api-access-9kg5s\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:31:22.793974 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:22.793970    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156801bb-2ae0-465d-a3b4-42b371fc8933-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:31:23.132666 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:23.132577    2575 generic.go:356] "Generic (PLEG): container finished" podID="156801bb-2ae0-465d-a3b4-42b371fc8933" containerID="5df9950af8399c6d110924cbd69055e953d883a8359cf9d1ba6bc023d663f27e" exitCode=0
Jul 28 17:31:23.132666 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:23.132621    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"
Jul 28 17:31:23.132859 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:23.132668    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq" event={"ID":"156801bb-2ae0-465d-a3b4-42b371fc8933","Type":"ContainerDied","Data":"5df9950af8399c6d110924cbd69055e953d883a8359cf9d1ba6bc023d663f27e"}
Jul 28 17:31:23.132859 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:23.132718    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq" event={"ID":"156801bb-2ae0-465d-a3b4-42b371fc8933","Type":"ContainerDied","Data":"edabe143c8161b06835517e81432c6c933f45f18b181b954ab92b815f363c6d8"}
Jul 28 17:31:23.132859 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:23.132742    2575 scope.go:117] "RemoveContainer" containerID="5df9950af8399c6d110924cbd69055e953d883a8359cf9d1ba6bc023d663f27e"
Jul 28 17:31:23.141448 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:23.141430    2575 scope.go:117] "RemoveContainer" containerID="5df9950af8399c6d110924cbd69055e953d883a8359cf9d1ba6bc023d663f27e"
Jul 28 17:31:23.141728 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:31:23.141707    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df9950af8399c6d110924cbd69055e953d883a8359cf9d1ba6bc023d663f27e\": container with ID starting with 5df9950af8399c6d110924cbd69055e953d883a8359cf9d1ba6bc023d663f27e not found: ID does not exist" containerID="5df9950af8399c6d110924cbd69055e953d883a8359cf9d1ba6bc023d663f27e"
Jul 28 17:31:23.141790 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:23.141737    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df9950af8399c6d110924cbd69055e953d883a8359cf9d1ba6bc023d663f27e"} err="failed to get container status \"5df9950af8399c6d110924cbd69055e953d883a8359cf9d1ba6bc023d663f27e\": rpc error: code = NotFound desc = could not find container \"5df9950af8399c6d110924cbd69055e953d883a8359cf9d1ba6bc023d663f27e\": container with ID starting with 5df9950af8399c6d110924cbd69055e953d883a8359cf9d1ba6bc023d663f27e not found: ID does not exist"
Jul 28 17:31:23.154598 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:23.154574    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"]
Jul 28 17:31:23.157961 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:23.157935    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-sm6sq"]
Jul 28 17:31:24.315544 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:24.315493    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="156801bb-2ae0-465d-a3b4-42b371fc8933" path="/var/lib/kubelet/pods/156801bb-2ae0-465d-a3b4-42b371fc8933/volumes"
Jul 28 17:31:27.742979 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.742944    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"]
Jul 28 17:31:27.743363 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.743283    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="156801bb-2ae0-465d-a3b4-42b371fc8933" containerName="main"
Jul 28 17:31:27.743363 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.743296    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="156801bb-2ae0-465d-a3b4-42b371fc8933" containerName="main"
Jul 28 17:31:27.743363 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.743361    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="156801bb-2ae0-465d-a3b4-42b371fc8933" containerName="main"
Jul 28 17:31:27.747711 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.747686    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.750408 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.750264    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Jul 28 17:31:27.750408 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.750306    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-hjbtl\""
Jul 28 17:31:27.750408 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.750331    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 28 17:31:27.750408 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.750354    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 28 17:31:27.759499 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.759472    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"]
Jul 28 17:31:27.836867 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.836830    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.836867 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.836867    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.837123 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.836896    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.837123 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.836956    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.837123 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.836978    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.837123 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.836998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.837123 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.837026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160e54fb-abfb-4915-9667-28c66d64e3f7-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.837440 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.837118    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.837440 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.837176    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb5cp\" (UniqueName: \"kubernetes.io/projected/160e54fb-abfb-4915-9667-28c66d64e3f7-kube-api-access-nb5cp\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.937951 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.937909    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb5cp\" (UniqueName: \"kubernetes.io/projected/160e54fb-abfb-4915-9667-28c66d64e3f7-kube-api-access-nb5cp\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.938144 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.937968    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.938144 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.937997    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.938144 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.938030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.938144 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.938073    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.938144 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.938098    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.938144 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.938133    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.938462 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.938178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160e54fb-abfb-4915-9667-28c66d64e3f7-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.938462 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.938224    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.938603 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.938499    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.938712 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.938690    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.938816 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.938735    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.938816 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.938784    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.939219 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.939199    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160e54fb-abfb-4915-9667-28c66d64e3f7-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.940681 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.940659    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.940772 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.940756    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.946982 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.946962    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:27.947198 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:27.947178    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb5cp\" (UniqueName: \"kubernetes.io/projected/160e54fb-abfb-4915-9667-28c66d64e3f7-kube-api-access-nb5cp\") pod \"router-gateway-1-openshift-default-5b4d9458bd-vlnql\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:28.059374 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:28.059287    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:28.209465 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:28.209440    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"]
Jul 28 17:31:28.211382 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:31:28.211350    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160e54fb_abfb_4915_9667_28c66d64e3f7.slice/crio-3d5c1ecac3145b38dadadd45d1d826fd1bec85e7c635af780058988d6de9468f WatchSource:0}: Error finding container 3d5c1ecac3145b38dadadd45d1d826fd1bec85e7c635af780058988d6de9468f: Status 404 returned error can't find the container with id 3d5c1ecac3145b38dadadd45d1d826fd1bec85e7c635af780058988d6de9468f
Jul 28 17:31:29.169087 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:29.165964    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql" event={"ID":"160e54fb-abfb-4915-9667-28c66d64e3f7","Type":"ContainerStarted","Data":"3d5c1ecac3145b38dadadd45d1d826fd1bec85e7c635af780058988d6de9468f"}
Jul 28 17:31:30.833060 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:30.833020    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 17:31:30.833336 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:30.833095    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 17:31:30.833336 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:30.833125    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 17:31:31.177599 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:31.177556    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql" event={"ID":"160e54fb-abfb-4915-9667-28c66d64e3f7","Type":"ContainerStarted","Data":"f9dd381439edffa75d49cca127e73240425e2a3588ee97cb996d55216aefc68e"}
Jul 28 17:31:31.200100 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:31.200053    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql" podStartSLOduration=1.580668648 podStartE2EDuration="4.20003781s" totalImagesPullingTime="2.619369162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:31:31.197600099 +0000 UTC m=+887.455209790" watchObservedRunningTime="2026-07-28 17:31:31.20003781 +0000 UTC m=+887.457647499"
Jul 28 17:31:32.060024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:32.059984    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:32.064536 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:32.064496    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:32.182644 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:32.182613    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:32.183676 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:32.183653    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:31:44.220499 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:44.220423    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:31:44.220906 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:44.220748    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:31:56.372292 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.372258    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"]
Jul 28 17:31:56.377267 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.377234    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.381197 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.381175    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Jul 28 17:31:56.381353 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.381173    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 28 17:31:56.382093 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.381995    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-8tncm\""
Jul 28 17:31:56.382093 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.382071    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 28 17:31:56.414964 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.414925    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"]
Jul 28 17:31:56.487039 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.486947    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.487039 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.486977    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10058793-3576-48c1-932e-6c2b2593e016-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.487039 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.487006    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10058793-3576-48c1-932e-6c2b2593e016-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.487330 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.487104    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xgsz\" (UniqueName: \"kubernetes.io/projected/10058793-3576-48c1-932e-6c2b2593e016-kube-api-access-2xgsz\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.487330 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.487142    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.487330 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.487170    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10058793-3576-48c1-932e-6c2b2593e016-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.487330 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.487201    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.487330 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.487230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.487330 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.487250    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.587964 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.587926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xgsz\" (UniqueName: \"kubernetes.io/projected/10058793-3576-48c1-932e-6c2b2593e016-kube-api-access-2xgsz\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.588170 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.587973    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.588170 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.587996    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10058793-3576-48c1-932e-6c2b2593e016-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.588170 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.588022    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.588170 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.588046    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.588170 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.588064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.588170 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.588086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.588170 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.588107    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10058793-3576-48c1-932e-6c2b2593e016-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.588170 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.588150    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10058793-3576-48c1-932e-6c2b2593e016-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.588617 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.588475    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.588839 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.588815    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.589096 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.589033    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10058793-3576-48c1-932e-6c2b2593e016-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.589096 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.589069    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.589320 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.589298    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.590592 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.590571    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.590750 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.590732    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10058793-3576-48c1-932e-6c2b2593e016-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.604475 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.604444    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10058793-3576-48c1-932e-6c2b2593e016-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.604579 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.604563    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xgsz\" (UniqueName: \"kubernetes.io/projected/10058793-3576-48c1-932e-6c2b2593e016-kube-api-access-2xgsz\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cj6hf\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.688634 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.688558    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:56.857614 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.856753    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"]
Jul 28 17:31:56.862782 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:31:56.862754    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10058793_3576_48c1_932e_6c2b2593e016.slice/crio-9fe6325d48836950ab46de3b3d4a72834b7009b613efc832e9f9f48b35382479 WatchSource:0}: Error finding container 9fe6325d48836950ab46de3b3d4a72834b7009b613efc832e9f9f48b35382479: Status 404 returned error can't find the container with id 9fe6325d48836950ab46de3b3d4a72834b7009b613efc832e9f9f48b35382479
Jul 28 17:31:56.864923 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.864888    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 17:31:56.864992 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.864974    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 17:31:56.865032 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:56.865016    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 17:31:57.297192 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:57.297155    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf" event={"ID":"10058793-3576-48c1-932e-6c2b2593e016","Type":"ContainerStarted","Data":"90688247033c9caec5262fb9fc43563f710d81d68dd5af7e775381c211973a8f"}
Jul 28 17:31:57.297353 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:57.297208    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf" event={"ID":"10058793-3576-48c1-932e-6c2b2593e016","Type":"ContainerStarted","Data":"9fe6325d48836950ab46de3b3d4a72834b7009b613efc832e9f9f48b35382479"}
Jul 28 17:31:57.317920 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:57.317856    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf" podStartSLOduration=1.317834866 podStartE2EDuration="1.317834866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:31:57.315749974 +0000 UTC m=+913.573359663" watchObservedRunningTime="2026-07-28 17:31:57.317834866 +0000 UTC m=+913.575444550"
Jul 28 17:31:57.689590 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:57.689555    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:57.694954 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:57.694930    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:58.302253 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:58.302220    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:31:58.303136 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:31:58.303113    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:32:01.017677 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:01.017641    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"]
Jul 28 17:32:01.018099 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:01.017882    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql" podUID="160e54fb-abfb-4915-9667-28c66d64e3f7" containerName="istio-proxy" containerID="cri-o://f9dd381439edffa75d49cca127e73240425e2a3588ee97cb996d55216aefc68e" gracePeriod=30
Jul 28 17:32:03.224716 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:03.224668    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql" podUID="160e54fb-abfb-4915-9667-28c66d64e3f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 17:32:03.981757 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:03.981690    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:32:04.054028 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.053934    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160e54fb-abfb-4915-9667-28c66d64e3f7-istiod-ca-cert\") pod \"160e54fb-abfb-4915-9667-28c66d64e3f7\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") "
Jul 28 17:32:04.054028 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.053981    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-envoy\") pod \"160e54fb-abfb-4915-9667-28c66d64e3f7\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") "
Jul 28 17:32:04.054028 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.054014    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-credential-socket\") pod \"160e54fb-abfb-4915-9667-28c66d64e3f7\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") "
Jul 28 17:32:04.054306 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.054046    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-podinfo\") pod \"160e54fb-abfb-4915-9667-28c66d64e3f7\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") "
Jul 28 17:32:04.054306 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.054093    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-data\") pod \"160e54fb-abfb-4915-9667-28c66d64e3f7\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") "
Jul 28 17:32:04.054306 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.054125    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-workload-socket\") pod \"160e54fb-abfb-4915-9667-28c66d64e3f7\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") "
Jul 28 17:32:04.054306 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.054185    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-workload-certs\") pod \"160e54fb-abfb-4915-9667-28c66d64e3f7\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") "
Jul 28 17:32:04.054306 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.054226    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-token\") pod \"160e54fb-abfb-4915-9667-28c66d64e3f7\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") "
Jul 28 17:32:04.054306 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.054293    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb5cp\" (UniqueName: \"kubernetes.io/projected/160e54fb-abfb-4915-9667-28c66d64e3f7-kube-api-access-nb5cp\") pod \"160e54fb-abfb-4915-9667-28c66d64e3f7\" (UID: \"160e54fb-abfb-4915-9667-28c66d64e3f7\") "
Jul 28 17:32:04.054625 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.054331    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160e54fb-abfb-4915-9667-28c66d64e3f7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "160e54fb-abfb-4915-9667-28c66d64e3f7" (UID: "160e54fb-abfb-4915-9667-28c66d64e3f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:04.054625 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.054597    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160e54fb-abfb-4915-9667-28c66d64e3f7-istiod-ca-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:04.054625 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.054608    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-data" (OuterVolumeSpecName: "istio-data") pod "160e54fb-abfb-4915-9667-28c66d64e3f7" (UID: "160e54fb-abfb-4915-9667-28c66d64e3f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:04.055149 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.055120    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "160e54fb-abfb-4915-9667-28c66d64e3f7" (UID: "160e54fb-abfb-4915-9667-28c66d64e3f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:04.055686 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.055659    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "160e54fb-abfb-4915-9667-28c66d64e3f7" (UID: "160e54fb-abfb-4915-9667-28c66d64e3f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:04.055766 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.055688    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "160e54fb-abfb-4915-9667-28c66d64e3f7" (UID: "160e54fb-abfb-4915-9667-28c66d64e3f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:04.057637 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.057585    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160e54fb-abfb-4915-9667-28c66d64e3f7-kube-api-access-nb5cp" (OuterVolumeSpecName: "kube-api-access-nb5cp") pod "160e54fb-abfb-4915-9667-28c66d64e3f7" (UID: "160e54fb-abfb-4915-9667-28c66d64e3f7"). InnerVolumeSpecName "kube-api-access-nb5cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:04.057739 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.057719    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "160e54fb-abfb-4915-9667-28c66d64e3f7" (UID: "160e54fb-abfb-4915-9667-28c66d64e3f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:04.058002 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.057954    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "160e54fb-abfb-4915-9667-28c66d64e3f7" (UID: "160e54fb-abfb-4915-9667-28c66d64e3f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 17:32:04.058076 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.057994    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "160e54fb-abfb-4915-9667-28c66d64e3f7" (UID: "160e54fb-abfb-4915-9667-28c66d64e3f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:04.155441 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.155407    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-envoy\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:04.155441 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.155434    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-credential-socket\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:04.155441 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.155445    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-podinfo\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:04.155696 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.155454    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-data\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:04.155696 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.155463    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-workload-socket\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:04.155696 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.155471    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160e54fb-abfb-4915-9667-28c66d64e3f7-workload-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:04.155696 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.155479    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160e54fb-abfb-4915-9667-28c66d64e3f7-istio-token\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:04.155696 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.155486    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb5cp\" (UniqueName: \"kubernetes.io/projected/160e54fb-abfb-4915-9667-28c66d64e3f7-kube-api-access-nb5cp\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:04.338796 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.338761    2575 generic.go:356] "Generic (PLEG): container finished" podID="160e54fb-abfb-4915-9667-28c66d64e3f7" containerID="f9dd381439edffa75d49cca127e73240425e2a3588ee97cb996d55216aefc68e" exitCode=0
Jul 28 17:32:04.339171 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.338817    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"
Jul 28 17:32:04.339171 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.338824    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql" event={"ID":"160e54fb-abfb-4915-9667-28c66d64e3f7","Type":"ContainerDied","Data":"f9dd381439edffa75d49cca127e73240425e2a3588ee97cb996d55216aefc68e"}
Jul 28 17:32:04.339171 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.338886    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql" event={"ID":"160e54fb-abfb-4915-9667-28c66d64e3f7","Type":"ContainerDied","Data":"3d5c1ecac3145b38dadadd45d1d826fd1bec85e7c635af780058988d6de9468f"}
Jul 28 17:32:04.339171 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.338909    2575 scope.go:117] "RemoveContainer" containerID="f9dd381439edffa75d49cca127e73240425e2a3588ee97cb996d55216aefc68e"
Jul 28 17:32:04.347281 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.347257    2575 scope.go:117] "RemoveContainer" containerID="f9dd381439edffa75d49cca127e73240425e2a3588ee97cb996d55216aefc68e"
Jul 28 17:32:04.347540 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:32:04.347506    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9dd381439edffa75d49cca127e73240425e2a3588ee97cb996d55216aefc68e\": container with ID starting with f9dd381439edffa75d49cca127e73240425e2a3588ee97cb996d55216aefc68e not found: ID does not exist" containerID="f9dd381439edffa75d49cca127e73240425e2a3588ee97cb996d55216aefc68e"
Jul 28 17:32:04.347602 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.347548    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9dd381439edffa75d49cca127e73240425e2a3588ee97cb996d55216aefc68e"} err="failed to get container status \"f9dd381439edffa75d49cca127e73240425e2a3588ee97cb996d55216aefc68e\": rpc error: code = NotFound desc = could not find container \"f9dd381439edffa75d49cca127e73240425e2a3588ee97cb996d55216aefc68e\": container with ID starting with f9dd381439edffa75d49cca127e73240425e2a3588ee97cb996d55216aefc68e not found: ID does not exist"
Jul 28 17:32:04.358913 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.358889    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"]
Jul 28 17:32:04.364169 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:04.364141    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-vlnql"]
Jul 28 17:32:06.312484 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:06.312448    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="160e54fb-abfb-4915-9667-28c66d64e3f7" path="/var/lib/kubelet/pods/160e54fb-abfb-4915-9667-28c66d64e3f7/volumes"
Jul 28 17:32:20.114041 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:20.113945    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"]
Jul 28 17:32:20.114484 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:20.114213    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf" podUID="10058793-3576-48c1-932e-6c2b2593e016" containerName="istio-proxy" containerID="cri-o://90688247033c9caec5262fb9fc43563f710d81d68dd5af7e775381c211973a8f" gracePeriod=30
Jul 28 17:32:22.964889 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:22.964864    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:32:23.021224 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.021190    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10058793-3576-48c1-932e-6c2b2593e016-istiod-ca-cert\") pod \"10058793-3576-48c1-932e-6c2b2593e016\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") "
Jul 28 17:32:23.021380 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.021276    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-workload-certs\") pod \"10058793-3576-48c1-932e-6c2b2593e016\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") "
Jul 28 17:32:23.021380 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.021307    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-credential-socket\") pod \"10058793-3576-48c1-932e-6c2b2593e016\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") "
Jul 28 17:32:23.021380 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.021351    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10058793-3576-48c1-932e-6c2b2593e016-istio-token\") pod \"10058793-3576-48c1-932e-6c2b2593e016\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") "
Jul 28 17:32:23.021380 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.021377    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-workload-socket\") pod \"10058793-3576-48c1-932e-6c2b2593e016\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") "
Jul 28 17:32:23.021710 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.021670    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10058793-3576-48c1-932e-6c2b2593e016-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "10058793-3576-48c1-932e-6c2b2593e016" (UID: "10058793-3576-48c1-932e-6c2b2593e016"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:23.021829 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.021701    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "10058793-3576-48c1-932e-6c2b2593e016" (UID: "10058793-3576-48c1-932e-6c2b2593e016"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:23.021926 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.021823    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "10058793-3576-48c1-932e-6c2b2593e016" (UID: "10058793-3576-48c1-932e-6c2b2593e016"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:23.021926 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.021841    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "10058793-3576-48c1-932e-6c2b2593e016" (UID: "10058793-3576-48c1-932e-6c2b2593e016"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:23.022058 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.021615    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xgsz\" (UniqueName: \"kubernetes.io/projected/10058793-3576-48c1-932e-6c2b2593e016-kube-api-access-2xgsz\") pod \"10058793-3576-48c1-932e-6c2b2593e016\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") "
Jul 28 17:32:23.022058 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.021993    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-istio-envoy\") pod \"10058793-3576-48c1-932e-6c2b2593e016\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") "
Jul 28 17:32:23.022058 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.022023    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-istio-data\") pod \"10058793-3576-48c1-932e-6c2b2593e016\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") "
Jul 28 17:32:23.022298 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.022279    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10058793-3576-48c1-932e-6c2b2593e016-istiod-ca-cert\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:23.022430 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.022417    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-workload-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:23.022542 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.022512    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-credential-socket\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:23.022661 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.022641    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-workload-socket\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:23.022661 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.022413    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-istio-data" (OuterVolumeSpecName: "istio-data") pod "10058793-3576-48c1-932e-6c2b2593e016" (UID: "10058793-3576-48c1-932e-6c2b2593e016"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:23.023987 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.023953    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10058793-3576-48c1-932e-6c2b2593e016-kube-api-access-2xgsz" (OuterVolumeSpecName: "kube-api-access-2xgsz") pod "10058793-3576-48c1-932e-6c2b2593e016" (UID: "10058793-3576-48c1-932e-6c2b2593e016"). InnerVolumeSpecName "kube-api-access-2xgsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:23.024082 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.024061    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10058793-3576-48c1-932e-6c2b2593e016-istio-token" (OuterVolumeSpecName: "istio-token") pod "10058793-3576-48c1-932e-6c2b2593e016" (UID: "10058793-3576-48c1-932e-6c2b2593e016"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:23.024286 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.024263    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "10058793-3576-48c1-932e-6c2b2593e016" (UID: "10058793-3576-48c1-932e-6c2b2593e016"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:23.123729 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.123705    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10058793-3576-48c1-932e-6c2b2593e016-istio-podinfo\") pod \"10058793-3576-48c1-932e-6c2b2593e016\" (UID: \"10058793-3576-48c1-932e-6c2b2593e016\") "
Jul 28 17:32:23.123925 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.123914    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-istio-envoy\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:23.123974 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.123927    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10058793-3576-48c1-932e-6c2b2593e016-istio-data\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:23.123974 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.123935    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10058793-3576-48c1-932e-6c2b2593e016-istio-token\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:23.123974 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.123944    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xgsz\" (UniqueName: \"kubernetes.io/projected/10058793-3576-48c1-932e-6c2b2593e016-kube-api-access-2xgsz\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:23.125544 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.125504    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10058793-3576-48c1-932e-6c2b2593e016-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "10058793-3576-48c1-932e-6c2b2593e016" (UID: "10058793-3576-48c1-932e-6c2b2593e016"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 17:32:23.225003 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.224967    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10058793-3576-48c1-932e-6c2b2593e016-istio-podinfo\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:32:23.436831 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.436799    2575 generic.go:356] "Generic (PLEG): container finished" podID="10058793-3576-48c1-932e-6c2b2593e016" containerID="90688247033c9caec5262fb9fc43563f710d81d68dd5af7e775381c211973a8f" exitCode=0
Jul 28 17:32:23.437001 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.436846    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"
Jul 28 17:32:23.437001 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.436884    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf" event={"ID":"10058793-3576-48c1-932e-6c2b2593e016","Type":"ContainerDied","Data":"90688247033c9caec5262fb9fc43563f710d81d68dd5af7e775381c211973a8f"}
Jul 28 17:32:23.437001 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.436942    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf" event={"ID":"10058793-3576-48c1-932e-6c2b2593e016","Type":"ContainerDied","Data":"9fe6325d48836950ab46de3b3d4a72834b7009b613efc832e9f9f48b35382479"}
Jul 28 17:32:23.437001 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.436971    2575 scope.go:117] "RemoveContainer" containerID="90688247033c9caec5262fb9fc43563f710d81d68dd5af7e775381c211973a8f"
Jul 28 17:32:23.447276 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.447259    2575 scope.go:117] "RemoveContainer" containerID="90688247033c9caec5262fb9fc43563f710d81d68dd5af7e775381c211973a8f"
Jul 28 17:32:23.447509 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:32:23.447491    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90688247033c9caec5262fb9fc43563f710d81d68dd5af7e775381c211973a8f\": container with ID starting with 90688247033c9caec5262fb9fc43563f710d81d68dd5af7e775381c211973a8f not found: ID does not exist" containerID="90688247033c9caec5262fb9fc43563f710d81d68dd5af7e775381c211973a8f"
Jul 28 17:32:23.447614 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.447516    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90688247033c9caec5262fb9fc43563f710d81d68dd5af7e775381c211973a8f"} err="failed to get container status \"90688247033c9caec5262fb9fc43563f710d81d68dd5af7e775381c211973a8f\": rpc error: code = NotFound desc = could not find container \"90688247033c9caec5262fb9fc43563f710d81d68dd5af7e775381c211973a8f\": container with ID starting with 90688247033c9caec5262fb9fc43563f710d81d68dd5af7e775381c211973a8f not found: ID does not exist"
Jul 28 17:32:23.465936 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.459858    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"]
Jul 28 17:32:23.466063 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:23.465965    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cj6hf"]
Jul 28 17:32:24.312144 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:24.312108    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10058793-3576-48c1-932e-6c2b2593e016" path="/var/lib/kubelet/pods/10058793-3576-48c1-932e-6c2b2593e016/volumes"
Jul 28 17:32:24.985445 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:24.985411    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8"]
Jul 28 17:32:24.985793 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:24.985779    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="160e54fb-abfb-4915-9667-28c66d64e3f7" containerName="istio-proxy"
Jul 28 17:32:24.985840 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:24.985795    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="160e54fb-abfb-4915-9667-28c66d64e3f7" containerName="istio-proxy"
Jul 28 17:32:24.985840 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:24.985802    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10058793-3576-48c1-932e-6c2b2593e016" containerName="istio-proxy"
Jul 28 17:32:24.985840 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:24.985809    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="10058793-3576-48c1-932e-6c2b2593e016" containerName="istio-proxy"
Jul 28 17:32:24.985929 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:24.985876    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="10058793-3576-48c1-932e-6c2b2593e016" containerName="istio-proxy"
Jul 28 17:32:24.985929 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:24.985884    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="160e54fb-abfb-4915-9667-28c66d64e3f7" containerName="istio-proxy"
Jul 28 17:32:24.988557 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:24.988514    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8"
Jul 28 17:32:24.992790 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:24.992770    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Jul 28 17:32:24.992790 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:24.992788    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-xrt2x\""
Jul 28 17:32:24.992971 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:24.992863    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-jkgsn\""
Jul 28 17:32:24.992971 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:24.992916    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Jul 28 17:32:24.992971 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:24.992916    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 28 17:32:24.999189 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:24.999167    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8"]
Jul 28 17:32:25.040429 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:25.040401    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k7np\" (UniqueName: \"kubernetes.io/projected/af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc-kube-api-access-7k7np\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8\" (UID: \"af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8"
Jul 28 17:32:25.040560 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:25.040489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8\" (UID: \"af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8"
Jul 28 17:32:25.140944 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:25.140903    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8\" (UID: \"af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8"
Jul 28 17:32:25.141112 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:25.140990    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k7np\" (UniqueName: \"kubernetes.io/projected/af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc-kube-api-access-7k7np\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8\" (UID: \"af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8"
Jul 28 17:32:25.143512 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:25.143481    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8\" (UID: \"af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8"
Jul 28 17:32:25.148503 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:25.148480    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k7np\" (UniqueName: \"kubernetes.io/projected/af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc-kube-api-access-7k7np\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8\" (UID: \"af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8"
Jul 28 17:32:25.299891 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:25.299800    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8"
Jul 28 17:32:25.433356 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:25.433322    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8"]
Jul 28 17:32:25.436972 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:32:25.436940    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf04fa4a_00ab_4bd7_8cd9_61a7fc324cdc.slice/crio-3dfe67766dd5357e1fae6ca18171d9e6cbdbc87d0b4c0bafe81be024a10bdd27 WatchSource:0}: Error finding container 3dfe67766dd5357e1fae6ca18171d9e6cbdbc87d0b4c0bafe81be024a10bdd27: Status 404 returned error can't find the container with id 3dfe67766dd5357e1fae6ca18171d9e6cbdbc87d0b4c0bafe81be024a10bdd27
Jul 28 17:32:25.447541 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:25.447502    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8" event={"ID":"af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc","Type":"ContainerStarted","Data":"3dfe67766dd5357e1fae6ca18171d9e6cbdbc87d0b4c0bafe81be024a10bdd27"}
Jul 28 17:32:26.453797 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:26.453763    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8" event={"ID":"af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc","Type":"ContainerStarted","Data":"2e452ac0ebc40805ca1f6f293ec77e10bb8da89ea22f04a4bf42bbc6bb6036bf"}
Jul 28 17:32:26.454238 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:26.453884    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8"
Jul 28 17:32:26.474248 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:26.474201    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8" podStartSLOduration=2.474184514 podStartE2EDuration="2.474184514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:26.472089591 +0000 UTC m=+942.729699282" watchObservedRunningTime="2026-07-28 17:32:26.474184514 +0000 UTC m=+942.731794204"
Jul 28 17:32:57.461030 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:32:57.461001    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8"
Jul 28 17:36:44.246994 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:36:44.246956    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:36:44.249070 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:36:44.249046    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:41:44.273250 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:41:44.273218    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:41:44.277480 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:41:44.277455    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:44:28.544219 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:28.544183    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"]
Jul 28 17:44:28.547717 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:28.547685    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"
Jul 28 17:44:28.550913 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:28.550882    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-5t9b9\""
Jul 28 17:44:28.551052 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:28.550971    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 28 17:44:28.551119 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:28.551078    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-wqpph\""
Jul 28 17:44:28.551119 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:28.551084    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 28 17:44:28.551298 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:28.551279    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 28 17:44:28.569283 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:28.569241    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"]
Jul 28 17:44:28.600343 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:28.600296    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6djrh\" (UniqueName: \"kubernetes.io/projected/45bef012-cfa8-4caa-bf59-eb26be70d69e-kube-api-access-6djrh\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl\" (UID: \"45bef012-cfa8-4caa-bf59-eb26be70d69e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"
Jul 28 17:44:28.600765 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:28.600733    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45bef012-cfa8-4caa-bf59-eb26be70d69e-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl\" (UID: \"45bef012-cfa8-4caa-bf59-eb26be70d69e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"
Jul 28 17:44:28.701490 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:28.701448    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45bef012-cfa8-4caa-bf59-eb26be70d69e-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl\" (UID: \"45bef012-cfa8-4caa-bf59-eb26be70d69e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"
Jul 28 17:44:28.701710 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:28.701596    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6djrh\" (UniqueName: \"kubernetes.io/projected/45bef012-cfa8-4caa-bf59-eb26be70d69e-kube-api-access-6djrh\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl\" (UID: \"45bef012-cfa8-4caa-bf59-eb26be70d69e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"
Jul 28 17:44:28.704104 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:28.704071    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45bef012-cfa8-4caa-bf59-eb26be70d69e-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl\" (UID: \"45bef012-cfa8-4caa-bf59-eb26be70d69e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"
Jul 28 17:44:28.709694 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:28.709670    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6djrh\" (UniqueName: \"kubernetes.io/projected/45bef012-cfa8-4caa-bf59-eb26be70d69e-kube-api-access-6djrh\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl\" (UID: \"45bef012-cfa8-4caa-bf59-eb26be70d69e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"
Jul 28 17:44:28.862939 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:28.862846    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"
Jul 28 17:44:29.005540 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:29.005493    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"]
Jul 28 17:44:29.008316 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:44:29.008283    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bef012_cfa8_4caa_bf59_eb26be70d69e.slice/crio-f4bb4b4a7326bdde39d9220809893ddf64a8c8d72ff4da24107461a4760ca390 WatchSource:0}: Error finding container f4bb4b4a7326bdde39d9220809893ddf64a8c8d72ff4da24107461a4760ca390: Status 404 returned error can't find the container with id f4bb4b4a7326bdde39d9220809893ddf64a8c8d72ff4da24107461a4760ca390
Jul 28 17:44:29.821219 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:29.821180    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl" event={"ID":"45bef012-cfa8-4caa-bf59-eb26be70d69e","Type":"ContainerStarted","Data":"af3119e6bc208181135122e037a110b882c96fd02f691b6b0b2a6f9010f09c0f"}
Jul 28 17:44:29.821713 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:29.821243    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl" event={"ID":"45bef012-cfa8-4caa-bf59-eb26be70d69e","Type":"ContainerStarted","Data":"f4bb4b4a7326bdde39d9220809893ddf64a8c8d72ff4da24107461a4760ca390"}
Jul 28 17:44:29.821713 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:29.821282    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"
Jul 28 17:44:29.841382 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:44:29.841323    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl" podStartSLOduration=1.841308365 podStartE2EDuration="1.841308365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:29.840352611 +0000 UTC m=+1666.097962303" watchObservedRunningTime="2026-07-28 17:44:29.841308365 +0000 UTC m=+1666.098918053"
Jul 28 17:45:00.827985 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:45:00.827950    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"
Jul 28 17:46:44.303426 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:46:44.303392    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:46:44.307801 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:46:44.307776    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:47:16.853200 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:16.853164    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-auth-enabled-requires-token-d4ed2c08_auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hgkl8_af04fa4a-00ab-4bd7-8cd9-61a7fc324cdc/main/0.log"
Jul 28 17:47:28.520508 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.520466    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"]
Jul 28 17:47:28.523830 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.523810    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"
Jul 28 17:47:28.526145 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.526111    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9371a7e9-043d-4bc3-805a-ecd84ec7f14c-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg\" (UID: \"9371a7e9-043d-4bc3-805a-ecd84ec7f14c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"
Jul 28 17:47:28.526295 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.526187    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgbsn\" (UniqueName: \"kubernetes.io/projected/9371a7e9-043d-4bc3-805a-ecd84ec7f14c-kube-api-access-hgbsn\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg\" (UID: \"9371a7e9-043d-4bc3-805a-ecd84ec7f14c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"
Jul 28 17:47:28.526952 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.526923    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 28 17:47:28.527081 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.527039    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-j75r7\""
Jul 28 17:47:28.527081 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.527043    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 28 17:47:28.527185 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.527047    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 28 17:47:28.527185 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.527110    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-9mgjj\""
Jul 28 17:47:28.533912 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.533882    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"]
Jul 28 17:47:28.627555 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.627483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9371a7e9-043d-4bc3-805a-ecd84ec7f14c-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg\" (UID: \"9371a7e9-043d-4bc3-805a-ecd84ec7f14c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"
Jul 28 17:47:28.627753 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.627587    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgbsn\" (UniqueName: \"kubernetes.io/projected/9371a7e9-043d-4bc3-805a-ecd84ec7f14c-kube-api-access-hgbsn\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg\" (UID: \"9371a7e9-043d-4bc3-805a-ecd84ec7f14c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"
Jul 28 17:47:28.630333 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.630299    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9371a7e9-043d-4bc3-805a-ecd84ec7f14c-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg\" (UID: \"9371a7e9-043d-4bc3-805a-ecd84ec7f14c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"
Jul 28 17:47:28.636434 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.636396    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgbsn\" (UniqueName: \"kubernetes.io/projected/9371a7e9-043d-4bc3-805a-ecd84ec7f14c-kube-api-access-hgbsn\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg\" (UID: \"9371a7e9-043d-4bc3-805a-ecd84ec7f14c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"
Jul 28 17:47:28.839784 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.839685    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"
Jul 28 17:47:28.987381 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:28.987354    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"]
Jul 28 17:47:28.989874 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:47:28.989840    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9371a7e9_043d_4bc3_805a_ecd84ec7f14c.slice/crio-961d3df6bcde414cd384d5fe871a8f5b2c32ea548271f3fac2f6a8946c25e695 WatchSource:0}: Error finding container 961d3df6bcde414cd384d5fe871a8f5b2c32ea548271f3fac2f6a8946c25e695: Status 404 returned error can't find the container with id 961d3df6bcde414cd384d5fe871a8f5b2c32ea548271f3fac2f6a8946c25e695
Jul 28 17:47:29.684790 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:29.684742    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg" event={"ID":"9371a7e9-043d-4bc3-805a-ecd84ec7f14c","Type":"ContainerStarted","Data":"75ec98bdcf36f776f7701f53aea69554b80cebccec2d596fda94f04e318f4a0b"}
Jul 28 17:47:29.685232 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:29.684809    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg" event={"ID":"9371a7e9-043d-4bc3-805a-ecd84ec7f14c","Type":"ContainerStarted","Data":"961d3df6bcde414cd384d5fe871a8f5b2c32ea548271f3fac2f6a8946c25e695"}
Jul 28 17:47:29.685232 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:29.684851    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"
Jul 28 17:47:29.705630 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:29.705560    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg" podStartSLOduration=1.705539553 podStartE2EDuration="1.705539553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:29.704035489 +0000 UTC m=+1845.961645179" watchObservedRunningTime="2026-07-28 17:47:29.705539553 +0000 UTC m=+1845.963149243"
Jul 28 17:47:40.039891 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:40.039851    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"]
Jul 28 17:47:40.827107 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:40.827068    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl" podUID="45bef012-cfa8-4caa-bf59-eb26be70d69e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:47:50.827517 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:50.827475    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl" podUID="45bef012-cfa8-4caa-bf59-eb26be70d69e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:47:55.060922 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.060872    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl" podUID="45bef012-cfa8-4caa-bf59-eb26be70d69e" containerName="main" containerID="cri-o://af3119e6bc208181135122e037a110b882c96fd02f691b6b0b2a6f9010f09c0f" gracePeriod=45
Jul 28 17:47:55.321486 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.321414    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"
Jul 28 17:47:55.476668 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.476619    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6djrh\" (UniqueName: \"kubernetes.io/projected/45bef012-cfa8-4caa-bf59-eb26be70d69e-kube-api-access-6djrh\") pod \"45bef012-cfa8-4caa-bf59-eb26be70d69e\" (UID: \"45bef012-cfa8-4caa-bf59-eb26be70d69e\") "
Jul 28 17:47:55.476848 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.476722    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45bef012-cfa8-4caa-bf59-eb26be70d69e-tls-certs\") pod \"45bef012-cfa8-4caa-bf59-eb26be70d69e\" (UID: \"45bef012-cfa8-4caa-bf59-eb26be70d69e\") "
Jul 28 17:47:55.479054 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.479018    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bef012-cfa8-4caa-bf59-eb26be70d69e-kube-api-access-6djrh" (OuterVolumeSpecName: "kube-api-access-6djrh") pod "45bef012-cfa8-4caa-bf59-eb26be70d69e" (UID: "45bef012-cfa8-4caa-bf59-eb26be70d69e"). InnerVolumeSpecName "kube-api-access-6djrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:55.479192 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.479080    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45bef012-cfa8-4caa-bf59-eb26be70d69e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45bef012-cfa8-4caa-bf59-eb26be70d69e" (UID: "45bef012-cfa8-4caa-bf59-eb26be70d69e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:47:55.578136 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.578024    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45bef012-cfa8-4caa-bf59-eb26be70d69e-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:47:55.578136 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.578064    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6djrh\" (UniqueName: \"kubernetes.io/projected/45bef012-cfa8-4caa-bf59-eb26be70d69e-kube-api-access-6djrh\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:47:55.819632 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.819592    2575 generic.go:356] "Generic (PLEG): container finished" podID="45bef012-cfa8-4caa-bf59-eb26be70d69e" containerID="af3119e6bc208181135122e037a110b882c96fd02f691b6b0b2a6f9010f09c0f" exitCode=0
Jul 28 17:47:55.819825 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.819648    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"
Jul 28 17:47:55.819825 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.819656    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl" event={"ID":"45bef012-cfa8-4caa-bf59-eb26be70d69e","Type":"ContainerDied","Data":"af3119e6bc208181135122e037a110b882c96fd02f691b6b0b2a6f9010f09c0f"}
Jul 28 17:47:55.819825 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.819725    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl" event={"ID":"45bef012-cfa8-4caa-bf59-eb26be70d69e","Type":"ContainerDied","Data":"f4bb4b4a7326bdde39d9220809893ddf64a8c8d72ff4da24107461a4760ca390"}
Jul 28 17:47:55.819825 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.819746    2575 scope.go:117] "RemoveContainer" containerID="af3119e6bc208181135122e037a110b882c96fd02f691b6b0b2a6f9010f09c0f"
Jul 28 17:47:55.831766 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.831738    2575 scope.go:117] "RemoveContainer" containerID="af3119e6bc208181135122e037a110b882c96fd02f691b6b0b2a6f9010f09c0f"
Jul 28 17:47:55.832428 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:47:55.832404    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af3119e6bc208181135122e037a110b882c96fd02f691b6b0b2a6f9010f09c0f\": container with ID starting with af3119e6bc208181135122e037a110b882c96fd02f691b6b0b2a6f9010f09c0f not found: ID does not exist" containerID="af3119e6bc208181135122e037a110b882c96fd02f691b6b0b2a6f9010f09c0f"
Jul 28 17:47:55.832572 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.832436    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3119e6bc208181135122e037a110b882c96fd02f691b6b0b2a6f9010f09c0f"} err="failed to get container status \"af3119e6bc208181135122e037a110b882c96fd02f691b6b0b2a6f9010f09c0f\": rpc error: code = NotFound desc = could not find container \"af3119e6bc208181135122e037a110b882c96fd02f691b6b0b2a6f9010f09c0f\": container with ID starting with af3119e6bc208181135122e037a110b882c96fd02f691b6b0b2a6f9010f09c0f not found: ID does not exist"
Jul 28 17:47:55.842882 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.842833    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"]
Jul 28 17:47:55.844993 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:55.844965    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2vgjl"]
Jul 28 17:47:56.312658 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:47:56.312609    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45bef012-cfa8-4caa-bf59-eb26be70d69e" path="/var/lib/kubelet/pods/45bef012-cfa8-4caa-bf59-eb26be70d69e/volumes"
Jul 28 17:48:00.692092 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:00.692059    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"
Jul 28 17:48:36.822388 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:36.822294    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"]
Jul 28 17:48:36.822913 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:36.822891    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45bef012-cfa8-4caa-bf59-eb26be70d69e" containerName="main"
Jul 28 17:48:36.822985 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:36.822917    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="45bef012-cfa8-4caa-bf59-eb26be70d69e" containerName="main"
Jul 28 17:48:36.823033 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:36.823021    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="45bef012-cfa8-4caa-bf59-eb26be70d69e" containerName="main"
Jul 28 17:48:36.826612 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:36.826586    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"
Jul 28 17:48:36.830645 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:36.830615    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-jrtdk\""
Jul 28 17:48:36.830831 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:36.830812    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 28 17:48:36.830995 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:36.830977    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 28 17:48:36.831048 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:36.831015    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 28 17:48:36.831048 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:36.831020    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-8gp7x\""
Jul 28 17:48:36.838438 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:36.838407    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"]
Jul 28 17:48:36.963078 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:36.963034    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562849e5-ea91-40ff-af91-432d937f4b1b-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h\" (UID: \"562849e5-ea91-40ff-af91-432d937f4b1b\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"
Jul 28 17:48:36.963267 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:36.963102    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-427tz\" (UniqueName: \"kubernetes.io/projected/562849e5-ea91-40ff-af91-432d937f4b1b-kube-api-access-427tz\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h\" (UID: \"562849e5-ea91-40ff-af91-432d937f4b1b\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"
Jul 28 17:48:37.064446 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:37.064403    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-427tz\" (UniqueName: \"kubernetes.io/projected/562849e5-ea91-40ff-af91-432d937f4b1b-kube-api-access-427tz\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h\" (UID: \"562849e5-ea91-40ff-af91-432d937f4b1b\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"
Jul 28 17:48:37.064675 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:37.064497    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562849e5-ea91-40ff-af91-432d937f4b1b-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h\" (UID: \"562849e5-ea91-40ff-af91-432d937f4b1b\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"
Jul 28 17:48:37.067326 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:37.067287    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562849e5-ea91-40ff-af91-432d937f4b1b-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h\" (UID: \"562849e5-ea91-40ff-af91-432d937f4b1b\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"
Jul 28 17:48:37.073676 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:37.073606    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-427tz\" (UniqueName: \"kubernetes.io/projected/562849e5-ea91-40ff-af91-432d937f4b1b-kube-api-access-427tz\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h\" (UID: \"562849e5-ea91-40ff-af91-432d937f4b1b\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"
Jul 28 17:48:37.138823 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:37.138782    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"
Jul 28 17:48:37.284900 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:37.284871    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"]
Jul 28 17:48:37.287371 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:48:37.287335    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562849e5_ea91_40ff_af91_432d937f4b1b.slice/crio-9f8858f6a737db4add54eafb8c10fd70282b0d8126652377a27c661e1d8b6daa WatchSource:0}: Error finding container 9f8858f6a737db4add54eafb8c10fd70282b0d8126652377a27c661e1d8b6daa: Status 404 returned error can't find the container with id 9f8858f6a737db4add54eafb8c10fd70282b0d8126652377a27c661e1d8b6daa
Jul 28 17:48:38.030719 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:38.030682    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h" event={"ID":"562849e5-ea91-40ff-af91-432d937f4b1b","Type":"ContainerStarted","Data":"13814efb88fdd219f26fd40444bf99d7bbb4c6527db45b2ba14d90078762bc56"}
Jul 28 17:48:38.031144 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:38.030737    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h" event={"ID":"562849e5-ea91-40ff-af91-432d937f4b1b","Type":"ContainerStarted","Data":"9f8858f6a737db4add54eafb8c10fd70282b0d8126652377a27c661e1d8b6daa"}
Jul 28 17:48:38.031144 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:38.030829    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"
Jul 28 17:48:38.050567 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:48:38.050491    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h" podStartSLOduration=2.0504748 podStartE2EDuration="2.0504748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:48:38.047638257 +0000 UTC m=+1914.305247941" watchObservedRunningTime="2026-07-28 17:48:38.0504748 +0000 UTC m=+1914.308084489"
Jul 28 17:49:09.039142 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:09.039105    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"
Jul 28 17:49:10.547268 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:10.547229    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"]
Jul 28 17:49:19.038955 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:19.038906    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h" podUID="562849e5-ea91-40ff-af91-432d937f4b1b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:49:25.568046 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:25.567995    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h" podUID="562849e5-ea91-40ff-af91-432d937f4b1b" containerName="main" containerID="cri-o://13814efb88fdd219f26fd40444bf99d7bbb4c6527db45b2ba14d90078762bc56" gracePeriod=45
Jul 28 17:49:25.834198 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:25.834167    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"
Jul 28 17:49:25.925901 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:25.925859    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562849e5-ea91-40ff-af91-432d937f4b1b-tls-certs\") pod \"562849e5-ea91-40ff-af91-432d937f4b1b\" (UID: \"562849e5-ea91-40ff-af91-432d937f4b1b\") "
Jul 28 17:49:25.926127 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:25.925935    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-427tz\" (UniqueName: \"kubernetes.io/projected/562849e5-ea91-40ff-af91-432d937f4b1b-kube-api-access-427tz\") pod \"562849e5-ea91-40ff-af91-432d937f4b1b\" (UID: \"562849e5-ea91-40ff-af91-432d937f4b1b\") "
Jul 28 17:49:25.928246 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:25.928215    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/562849e5-ea91-40ff-af91-432d937f4b1b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "562849e5-ea91-40ff-af91-432d937f4b1b" (UID: "562849e5-ea91-40ff-af91-432d937f4b1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:49:25.928372 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:25.928345    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562849e5-ea91-40ff-af91-432d937f4b1b-kube-api-access-427tz" (OuterVolumeSpecName: "kube-api-access-427tz") pod "562849e5-ea91-40ff-af91-432d937f4b1b" (UID: "562849e5-ea91-40ff-af91-432d937f4b1b"). InnerVolumeSpecName "kube-api-access-427tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:49:26.026510 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:26.026473    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562849e5-ea91-40ff-af91-432d937f4b1b-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:49:26.026510 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:26.026506    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-427tz\" (UniqueName: \"kubernetes.io/projected/562849e5-ea91-40ff-af91-432d937f4b1b-kube-api-access-427tz\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:49:26.269568 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:26.269500    2575 generic.go:356] "Generic (PLEG): container finished" podID="562849e5-ea91-40ff-af91-432d937f4b1b" containerID="13814efb88fdd219f26fd40444bf99d7bbb4c6527db45b2ba14d90078762bc56" exitCode=0
Jul 28 17:49:26.269568 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:26.269557    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"
Jul 28 17:49:26.269788 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:26.269602    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h" event={"ID":"562849e5-ea91-40ff-af91-432d937f4b1b","Type":"ContainerDied","Data":"13814efb88fdd219f26fd40444bf99d7bbb4c6527db45b2ba14d90078762bc56"}
Jul 28 17:49:26.269788 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:26.269660    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h" event={"ID":"562849e5-ea91-40ff-af91-432d937f4b1b","Type":"ContainerDied","Data":"9f8858f6a737db4add54eafb8c10fd70282b0d8126652377a27c661e1d8b6daa"}
Jul 28 17:49:26.269788 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:26.269683    2575 scope.go:117] "RemoveContainer" containerID="13814efb88fdd219f26fd40444bf99d7bbb4c6527db45b2ba14d90078762bc56"
Jul 28 17:49:26.278871 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:26.278850    2575 scope.go:117] "RemoveContainer" containerID="13814efb88fdd219f26fd40444bf99d7bbb4c6527db45b2ba14d90078762bc56"
Jul 28 17:49:26.279175 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:49:26.279155    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13814efb88fdd219f26fd40444bf99d7bbb4c6527db45b2ba14d90078762bc56\": container with ID starting with 13814efb88fdd219f26fd40444bf99d7bbb4c6527db45b2ba14d90078762bc56 not found: ID does not exist" containerID="13814efb88fdd219f26fd40444bf99d7bbb4c6527db45b2ba14d90078762bc56"
Jul 28 17:49:26.279229 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:26.279184    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13814efb88fdd219f26fd40444bf99d7bbb4c6527db45b2ba14d90078762bc56"} err="failed to get container status \"13814efb88fdd219f26fd40444bf99d7bbb4c6527db45b2ba14d90078762bc56\": rpc error: code = NotFound desc = could not find container \"13814efb88fdd219f26fd40444bf99d7bbb4c6527db45b2ba14d90078762bc56\": container with ID starting with 13814efb88fdd219f26fd40444bf99d7bbb4c6527db45b2ba14d90078762bc56 not found: ID does not exist"
Jul 28 17:49:26.290992 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:26.290951    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"]
Jul 28 17:49:26.294504 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:26.294472    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5kss7h"]
Jul 28 17:49:26.312930 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:26.312897    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="562849e5-ea91-40ff-af91-432d937f4b1b" path="/var/lib/kubelet/pods/562849e5-ea91-40ff-af91-432d937f4b1b/volumes"
Jul 28 17:49:30.692090 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:30.692045    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg" podUID="9371a7e9-043d-4bc3-805a-ecd84ec7f14c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:49:30.740260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:30.740218    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"]
Jul 28 17:49:40.692645 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:40.692596    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg" podUID="9371a7e9-043d-4bc3-805a-ecd84ec7f14c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:49:45.762732 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:45.762683    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg" podUID="9371a7e9-043d-4bc3-805a-ecd84ec7f14c" containerName="main" containerID="cri-o://75ec98bdcf36f776f7701f53aea69554b80cebccec2d596fda94f04e318f4a0b" gracePeriod=45
Jul 28 17:49:45.806467 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:49:45.806421    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9371a7e9_043d_4bc3_805a_ecd84ec7f14c.slice/crio-75ec98bdcf36f776f7701f53aea69554b80cebccec2d596fda94f04e318f4a0b.scope\": RecentStats: unable to find data in memory cache]"
Jul 28 17:49:46.032846 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.032814    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"
Jul 28 17:49:46.096671 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.096627    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgbsn\" (UniqueName: \"kubernetes.io/projected/9371a7e9-043d-4bc3-805a-ecd84ec7f14c-kube-api-access-hgbsn\") pod \"9371a7e9-043d-4bc3-805a-ecd84ec7f14c\" (UID: \"9371a7e9-043d-4bc3-805a-ecd84ec7f14c\") "
Jul 28 17:49:46.096885 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.096699    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9371a7e9-043d-4bc3-805a-ecd84ec7f14c-tls-certs\") pod \"9371a7e9-043d-4bc3-805a-ecd84ec7f14c\" (UID: \"9371a7e9-043d-4bc3-805a-ecd84ec7f14c\") "
Jul 28 17:49:46.098643 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.098610    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"]
Jul 28 17:49:46.099003 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.098979    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9371a7e9-043d-4bc3-805a-ecd84ec7f14c" containerName="main"
Jul 28 17:49:46.099003 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.099001    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9371a7e9-043d-4bc3-805a-ecd84ec7f14c" containerName="main"
Jul 28 17:49:46.099184 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.099012    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="562849e5-ea91-40ff-af91-432d937f4b1b" containerName="main"
Jul 28 17:49:46.099184 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.099021    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="562849e5-ea91-40ff-af91-432d937f4b1b" containerName="main"
Jul 28 17:49:46.099184 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.099126    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9371a7e9-043d-4bc3-805a-ecd84ec7f14c" containerName="main"
Jul 28 17:49:46.099184 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.099140    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="562849e5-ea91-40ff-af91-432d937f4b1b" containerName="main"
Jul 28 17:49:46.099347 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.099311    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9371a7e9-043d-4bc3-805a-ecd84ec7f14c-kube-api-access-hgbsn" (OuterVolumeSpecName: "kube-api-access-hgbsn") pod "9371a7e9-043d-4bc3-805a-ecd84ec7f14c" (UID: "9371a7e9-043d-4bc3-805a-ecd84ec7f14c"). InnerVolumeSpecName "kube-api-access-hgbsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:49:46.099384 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.099358    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9371a7e9-043d-4bc3-805a-ecd84ec7f14c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9371a7e9-043d-4bc3-805a-ecd84ec7f14c" (UID: "9371a7e9-043d-4bc3-805a-ecd84ec7f14c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:49:46.102052 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.102035    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.104293 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.104273    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 28 17:49:46.104293 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.104284    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-8b47b\""
Jul 28 17:49:46.104450 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.104301    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 28 17:49:46.104450 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.104288    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 28 17:49:46.112658 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.112628    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"]
Jul 28 17:49:46.198142 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.198092    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.198142 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.198145    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gfdp\" (UniqueName: \"kubernetes.io/projected/07c438b5-ca93-4904-81c5-4e134ae4bf53-kube-api-access-2gfdp\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.198367 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.198246    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.198367 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.198298    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.198367 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.198330    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.198462 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.198382    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c438b5-ca93-4904-81c5-4e134ae4bf53-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.198462 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.198453    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgbsn\" (UniqueName: \"kubernetes.io/projected/9371a7e9-043d-4bc3-805a-ecd84ec7f14c-kube-api-access-hgbsn\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:49:46.198543 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.198464    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9371a7e9-043d-4bc3-805a-ecd84ec7f14c-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:49:46.299396 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.299277    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.299396 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.299340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.299396 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.299390    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c438b5-ca93-4904-81c5-4e134ae4bf53-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.299732 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.299432    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.299732 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.299456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gfdp\" (UniqueName: \"kubernetes.io/projected/07c438b5-ca93-4904-81c5-4e134ae4bf53-kube-api-access-2gfdp\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.299732 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.299517    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.299841 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.299812    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.299881 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.299842    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.299924 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.299886    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.301870 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.301844    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.302193 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.302175    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c438b5-ca93-4904-81c5-4e134ae4bf53-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.307570 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.307546    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gfdp\" (UniqueName: \"kubernetes.io/projected/07c438b5-ca93-4904-81c5-4e134ae4bf53-kube-api-access-2gfdp\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.371583 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.371540    2575 generic.go:356] "Generic (PLEG): container finished" podID="9371a7e9-043d-4bc3-805a-ecd84ec7f14c" containerID="75ec98bdcf36f776f7701f53aea69554b80cebccec2d596fda94f04e318f4a0b" exitCode=0
Jul 28 17:49:46.371785 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.371590    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"
Jul 28 17:49:46.371785 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.371623    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg" event={"ID":"9371a7e9-043d-4bc3-805a-ecd84ec7f14c","Type":"ContainerDied","Data":"75ec98bdcf36f776f7701f53aea69554b80cebccec2d596fda94f04e318f4a0b"}
Jul 28 17:49:46.371785 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.371691    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg" event={"ID":"9371a7e9-043d-4bc3-805a-ecd84ec7f14c","Type":"ContainerDied","Data":"961d3df6bcde414cd384d5fe871a8f5b2c32ea548271f3fac2f6a8946c25e695"}
Jul 28 17:49:46.371785 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.371721    2575 scope.go:117] "RemoveContainer" containerID="75ec98bdcf36f776f7701f53aea69554b80cebccec2d596fda94f04e318f4a0b"
Jul 28 17:49:46.382954 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.382922    2575 scope.go:117] "RemoveContainer" containerID="75ec98bdcf36f776f7701f53aea69554b80cebccec2d596fda94f04e318f4a0b"
Jul 28 17:49:46.383287 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:49:46.383263    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ec98bdcf36f776f7701f53aea69554b80cebccec2d596fda94f04e318f4a0b\": container with ID starting with 75ec98bdcf36f776f7701f53aea69554b80cebccec2d596fda94f04e318f4a0b not found: ID does not exist" containerID="75ec98bdcf36f776f7701f53aea69554b80cebccec2d596fda94f04e318f4a0b"
Jul 28 17:49:46.383343 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.383298    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ec98bdcf36f776f7701f53aea69554b80cebccec2d596fda94f04e318f4a0b"} err="failed to get container status \"75ec98bdcf36f776f7701f53aea69554b80cebccec2d596fda94f04e318f4a0b\": rpc error: code = NotFound desc = could not find container \"75ec98bdcf36f776f7701f53aea69554b80cebccec2d596fda94f04e318f4a0b\": container with ID starting with 75ec98bdcf36f776f7701f53aea69554b80cebccec2d596fda94f04e318f4a0b not found: ID does not exist"
Jul 28 17:49:46.386690 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.386662    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"]
Jul 28 17:49:46.390799 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.390772    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcm6ktg"]
Jul 28 17:49:46.424053 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.424021    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:46.571230 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.571133    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"]
Jul 28 17:49:46.574500 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:49:46.574471    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c438b5_ca93_4904_81c5_4e134ae4bf53.slice/crio-7908963290b7cf04c558aaaa4f620ec0fecfe5e6b400a39b2ca309dac104ac87 WatchSource:0}: Error finding container 7908963290b7cf04c558aaaa4f620ec0fecfe5e6b400a39b2ca309dac104ac87: Status 404 returned error can't find the container with id 7908963290b7cf04c558aaaa4f620ec0fecfe5e6b400a39b2ca309dac104ac87
Jul 28 17:49:46.576355 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:46.576331    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:49:47.382334 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:47.382273    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v" event={"ID":"07c438b5-ca93-4904-81c5-4e134ae4bf53","Type":"ContainerStarted","Data":"7908963290b7cf04c558aaaa4f620ec0fecfe5e6b400a39b2ca309dac104ac87"}
Jul 28 17:49:48.313369 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:48.313333    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9371a7e9-043d-4bc3-805a-ecd84ec7f14c" path="/var/lib/kubelet/pods/9371a7e9-043d-4bc3-805a-ecd84ec7f14c/volumes"
Jul 28 17:49:48.388964 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:48.388920    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v" event={"ID":"07c438b5-ca93-4904-81c5-4e134ae4bf53","Type":"ContainerStarted","Data":"1e0ef87de4e2b287cbdbda3eb11551ec877c91bedd3445bd8eea17e8ca95cc2b"}
Jul 28 17:49:48.409069 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:48.409006    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v" podStartSLOduration=0.700413433 podStartE2EDuration="2.408986548s" totalImagesPullingTime="1.708573115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:49:48.407913219 +0000 UTC m=+1984.665522909" watchObservedRunningTime="2026-07-28 17:49:48.408986548 +0000 UTC m=+1984.666596237"
Jul 28 17:49:48.424812 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:48.424776    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:56.424270 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:56.424233    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:56.437290 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:56.437258    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:49:57.435729 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:49:57.435692    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:50:14.259218 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.259178    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"]
Jul 28 17:50:14.267114 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.267085    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"
Jul 28 17:50:14.269538 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.269482    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 28 17:50:14.270151 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.270124    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-bhsp7\""
Jul 28 17:50:14.270308 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.270183    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-wffsh\""
Jul 28 17:50:14.270308 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.270129    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 28 17:50:14.270308 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.270126    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 28 17:50:14.270992 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.270970    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"]
Jul 28 17:50:14.362978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.362936    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9471a569-4746-4ed0-882e-b2fc4f485d0e-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm\" (UID: \"9471a569-4746-4ed0-882e-b2fc4f485d0e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"
Jul 28 17:50:14.363209 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.362991    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5dh\" (UniqueName: \"kubernetes.io/projected/9471a569-4746-4ed0-882e-b2fc4f485d0e-kube-api-access-6f5dh\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm\" (UID: \"9471a569-4746-4ed0-882e-b2fc4f485d0e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"
Jul 28 17:50:14.464192 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.464151    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9471a569-4746-4ed0-882e-b2fc4f485d0e-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm\" (UID: \"9471a569-4746-4ed0-882e-b2fc4f485d0e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"
Jul 28 17:50:14.464393 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.464207    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5dh\" (UniqueName: \"kubernetes.io/projected/9471a569-4746-4ed0-882e-b2fc4f485d0e-kube-api-access-6f5dh\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm\" (UID: \"9471a569-4746-4ed0-882e-b2fc4f485d0e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"
Jul 28 17:50:14.467026 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.466994    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9471a569-4746-4ed0-882e-b2fc4f485d0e-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm\" (UID: \"9471a569-4746-4ed0-882e-b2fc4f485d0e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"
Jul 28 17:50:14.472221 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.472190    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5dh\" (UniqueName: \"kubernetes.io/projected/9471a569-4746-4ed0-882e-b2fc4f485d0e-kube-api-access-6f5dh\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm\" (UID: \"9471a569-4746-4ed0-882e-b2fc4f485d0e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"
Jul 28 17:50:14.580594 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.580484    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"
Jul 28 17:50:14.727709 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:14.727677    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"]
Jul 28 17:50:14.729920 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:50:14.729892    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9471a569_4746_4ed0_882e_b2fc4f485d0e.slice/crio-34b647db7d98e2e61b8720e600e62deedaaaf883a3c3d008908a5d50ee5b771b WatchSource:0}: Error finding container 34b647db7d98e2e61b8720e600e62deedaaaf883a3c3d008908a5d50ee5b771b: Status 404 returned error can't find the container with id 34b647db7d98e2e61b8720e600e62deedaaaf883a3c3d008908a5d50ee5b771b
Jul 28 17:50:15.522298 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:15.522257    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm" event={"ID":"9471a569-4746-4ed0-882e-b2fc4f485d0e","Type":"ContainerStarted","Data":"0394493a17a9e2f33531a009feb565d6f25562c817345914426a329434be3ae5"}
Jul 28 17:50:15.522726 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:15.522327    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm" event={"ID":"9471a569-4746-4ed0-882e-b2fc4f485d0e","Type":"ContainerStarted","Data":"34b647db7d98e2e61b8720e600e62deedaaaf883a3c3d008908a5d50ee5b771b"}
Jul 28 17:50:15.522726 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:15.522447    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"
Jul 28 17:50:15.540146 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:15.540084    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm" podStartSLOduration=1.540063007 podStartE2EDuration="1.540063007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:50:15.538091501 +0000 UTC m=+2011.795701190" watchObservedRunningTime="2026-07-28 17:50:15.540063007 +0000 UTC m=+2011.797672697"
Jul 28 17:50:37.087308 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.087266    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"]
Jul 28 17:50:37.096929 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:50:37.096886    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:50:37.096929 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:50:37.096929 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 17:50:37.096929 ip-10-0-135-188 kubenswrapper[2575]:  > containerID="1e0ef87de4e2b287cbdbda3eb11551ec877c91bedd3445bd8eea17e8ca95cc2b" cmd=["/bin/sleep","15"]
Jul 28 17:50:37.097258 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.096953    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v" message="" err=<
Jul 28 17:50:37.097258 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:50:37.097258 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 17:50:37.097258 ip-10-0-135-188 kubenswrapper[2575]:  >
Jul 28 17:50:37.097258 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:50:37.096970    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:50:37.097258 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:50:37.097258 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 17:50:37.097258 ip-10-0-135-188 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v" podUID="07c438b5-ca93-4904-81c5-4e134ae4bf53" containerName="main" containerID="cri-o://1e0ef87de4e2b287cbdbda3eb11551ec877c91bedd3445bd8eea17e8ca95cc2b"
Jul 28 17:50:37.097258 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.097017    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v" podUID="07c438b5-ca93-4904-81c5-4e134ae4bf53" containerName="main" containerID="cri-o://1e0ef87de4e2b287cbdbda3eb11551ec877c91bedd3445bd8eea17e8ca95cc2b" gracePeriod=60
Jul 28 17:50:37.456699 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.456664    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:50:37.465047 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.465015    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c438b5-ca93-4904-81c5-4e134ae4bf53-tls-certs\") pod \"07c438b5-ca93-4904-81c5-4e134ae4bf53\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") "
Jul 28 17:50:37.465226 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.465051    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-home\") pod \"07c438b5-ca93-4904-81c5-4e134ae4bf53\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") "
Jul 28 17:50:37.465226 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.465096    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-dshm\") pod \"07c438b5-ca93-4904-81c5-4e134ae4bf53\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") "
Jul 28 17:50:37.465226 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.465151    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-model-cache\") pod \"07c438b5-ca93-4904-81c5-4e134ae4bf53\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") "
Jul 28 17:50:37.465226 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.465168    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-tmp-dir\") pod \"07c438b5-ca93-4904-81c5-4e134ae4bf53\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") "
Jul 28 17:50:37.465226 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.465225    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gfdp\" (UniqueName: \"kubernetes.io/projected/07c438b5-ca93-4904-81c5-4e134ae4bf53-kube-api-access-2gfdp\") pod \"07c438b5-ca93-4904-81c5-4e134ae4bf53\" (UID: \"07c438b5-ca93-4904-81c5-4e134ae4bf53\") "
Jul 28 17:50:37.465493 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.465409    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-model-cache" (OuterVolumeSpecName: "model-cache") pod "07c438b5-ca93-4904-81c5-4e134ae4bf53" (UID: "07c438b5-ca93-4904-81c5-4e134ae4bf53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:50:37.465493 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.465423    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-home" (OuterVolumeSpecName: "home") pod "07c438b5-ca93-4904-81c5-4e134ae4bf53" (UID: "07c438b5-ca93-4904-81c5-4e134ae4bf53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:50:37.465493 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.465455    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07c438b5-ca93-4904-81c5-4e134ae4bf53" (UID: "07c438b5-ca93-4904-81c5-4e134ae4bf53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:50:37.467415 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.467388    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-dshm" (OuterVolumeSpecName: "dshm") pod "07c438b5-ca93-4904-81c5-4e134ae4bf53" (UID: "07c438b5-ca93-4904-81c5-4e134ae4bf53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:50:37.467602 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.467429    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07c438b5-ca93-4904-81c5-4e134ae4bf53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07c438b5-ca93-4904-81c5-4e134ae4bf53" (UID: "07c438b5-ca93-4904-81c5-4e134ae4bf53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:50:37.467602 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.467557    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c438b5-ca93-4904-81c5-4e134ae4bf53-kube-api-access-2gfdp" (OuterVolumeSpecName: "kube-api-access-2gfdp") pod "07c438b5-ca93-4904-81c5-4e134ae4bf53" (UID: "07c438b5-ca93-4904-81c5-4e134ae4bf53"). InnerVolumeSpecName "kube-api-access-2gfdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:50:37.566155 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.566095    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-dshm\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:50:37.566155 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.566147    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-model-cache\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:50:37.566155 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.566158    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-tmp-dir\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:50:37.566155 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.566169    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gfdp\" (UniqueName: \"kubernetes.io/projected/07c438b5-ca93-4904-81c5-4e134ae4bf53-kube-api-access-2gfdp\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:50:37.566802 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.566179    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c438b5-ca93-4904-81c5-4e134ae4bf53-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:50:37.566802 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.566187    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c438b5-ca93-4904-81c5-4e134ae4bf53-home\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:50:37.631001 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.630907    2575 generic.go:356] "Generic (PLEG): container finished" podID="07c438b5-ca93-4904-81c5-4e134ae4bf53" containerID="1e0ef87de4e2b287cbdbda3eb11551ec877c91bedd3445bd8eea17e8ca95cc2b" exitCode=0
Jul 28 17:50:37.631001 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.630961    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"
Jul 28 17:50:37.631194 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.630995    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v" event={"ID":"07c438b5-ca93-4904-81c5-4e134ae4bf53","Type":"ContainerDied","Data":"1e0ef87de4e2b287cbdbda3eb11551ec877c91bedd3445bd8eea17e8ca95cc2b"}
Jul 28 17:50:37.631194 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.631056    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v" event={"ID":"07c438b5-ca93-4904-81c5-4e134ae4bf53","Type":"ContainerDied","Data":"7908963290b7cf04c558aaaa4f620ec0fecfe5e6b400a39b2ca309dac104ac87"}
Jul 28 17:50:37.631194 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.631085    2575 scope.go:117] "RemoveContainer" containerID="1e0ef87de4e2b287cbdbda3eb11551ec877c91bedd3445bd8eea17e8ca95cc2b"
Jul 28 17:50:37.641957 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.641932    2575 scope.go:117] "RemoveContainer" containerID="1e0ef87de4e2b287cbdbda3eb11551ec877c91bedd3445bd8eea17e8ca95cc2b"
Jul 28 17:50:37.642295 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:50:37.642266    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e0ef87de4e2b287cbdbda3eb11551ec877c91bedd3445bd8eea17e8ca95cc2b\": container with ID starting with 1e0ef87de4e2b287cbdbda3eb11551ec877c91bedd3445bd8eea17e8ca95cc2b not found: ID does not exist" containerID="1e0ef87de4e2b287cbdbda3eb11551ec877c91bedd3445bd8eea17e8ca95cc2b"
Jul 28 17:50:37.642414 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.642302    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0ef87de4e2b287cbdbda3eb11551ec877c91bedd3445bd8eea17e8ca95cc2b"} err="failed to get container status \"1e0ef87de4e2b287cbdbda3eb11551ec877c91bedd3445bd8eea17e8ca95cc2b\": rpc error: code = NotFound desc = could not find container \"1e0ef87de4e2b287cbdbda3eb11551ec877c91bedd3445bd8eea17e8ca95cc2b\": container with ID starting with 1e0ef87de4e2b287cbdbda3eb11551ec877c91bedd3445bd8eea17e8ca95cc2b not found: ID does not exist"
Jul 28 17:50:37.669218 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.669181    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"]
Jul 28 17:50:37.680363 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:37.680327    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v"]
Jul 28 17:50:38.312470 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:38.312430    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07c438b5-ca93-4904-81c5-4e134ae4bf53" path="/var/lib/kubelet/pods/07c438b5-ca93-4904-81c5-4e134ae4bf53/volumes"
Jul 28 17:50:38.437197 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:38.437139    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f64z2v" podUID="07c438b5-ca93-4904-81c5-4e134ae4bf53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": context deadline exceeded"
Jul 28 17:50:46.528253 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:50:46.528222    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"
Jul 28 17:51:44.332182 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:51:44.332147    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:51:44.338282 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:51:44.338253    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:52:09.713692 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.713652    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"]
Jul 28 17:52:09.714099 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.714013    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c438b5-ca93-4904-81c5-4e134ae4bf53" containerName="main"
Jul 28 17:52:09.714099 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.714027    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c438b5-ca93-4904-81c5-4e134ae4bf53" containerName="main"
Jul 28 17:52:09.714099 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.714093    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="07c438b5-ca93-4904-81c5-4e134ae4bf53" containerName="main"
Jul 28 17:52:09.717219 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.717193    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"
Jul 28 17:52:09.719407 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.719377    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-v568p\""
Jul 28 17:52:09.720120 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.720102    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-hdmxk\""
Jul 28 17:52:09.720247 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.720230    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 28 17:52:09.720289 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.720229    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 28 17:52:09.720337 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.720328    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 28 17:52:09.726966 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.726930    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"]
Jul 28 17:52:09.799005 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.798960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e97b04b-b1e9-4d77-b46b-acce452b64d3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2\" (UID: \"5e97b04b-b1e9-4d77-b46b-acce452b64d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"
Jul 28 17:52:09.799191 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.799026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckgc6\" (UniqueName: \"kubernetes.io/projected/5e97b04b-b1e9-4d77-b46b-acce452b64d3-kube-api-access-ckgc6\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2\" (UID: \"5e97b04b-b1e9-4d77-b46b-acce452b64d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"
Jul 28 17:52:09.900091 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.900048    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e97b04b-b1e9-4d77-b46b-acce452b64d3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2\" (UID: \"5e97b04b-b1e9-4d77-b46b-acce452b64d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"
Jul 28 17:52:09.900324 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.900109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckgc6\" (UniqueName: \"kubernetes.io/projected/5e97b04b-b1e9-4d77-b46b-acce452b64d3-kube-api-access-ckgc6\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2\" (UID: \"5e97b04b-b1e9-4d77-b46b-acce452b64d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"
Jul 28 17:52:09.902684 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.902656    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e97b04b-b1e9-4d77-b46b-acce452b64d3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2\" (UID: \"5e97b04b-b1e9-4d77-b46b-acce452b64d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"
Jul 28 17:52:09.909317 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:09.909291    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckgc6\" (UniqueName: \"kubernetes.io/projected/5e97b04b-b1e9-4d77-b46b-acce452b64d3-kube-api-access-ckgc6\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2\" (UID: \"5e97b04b-b1e9-4d77-b46b-acce452b64d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"
Jul 28 17:52:10.029237 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:10.029138    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"
Jul 28 17:52:10.174667 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:10.174640    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"]
Jul 28 17:52:10.177532 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:52:10.177488    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e97b04b_b1e9_4d77_b46b_acce452b64d3.slice/crio-f999dabed223234246fe8cfc39c26d429668f5542d98658cbf04f538d6fc8fdb WatchSource:0}: Error finding container f999dabed223234246fe8cfc39c26d429668f5542d98658cbf04f538d6fc8fdb: Status 404 returned error can't find the container with id f999dabed223234246fe8cfc39c26d429668f5542d98658cbf04f538d6fc8fdb
Jul 28 17:52:11.075956 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:11.075910    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2" event={"ID":"5e97b04b-b1e9-4d77-b46b-acce452b64d3","Type":"ContainerStarted","Data":"e985644e880ba945851fffaa6330bd512a4c85feb7c38e00736528520df0b358"}
Jul 28 17:52:11.076409 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:11.075975    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2" event={"ID":"5e97b04b-b1e9-4d77-b46b-acce452b64d3","Type":"ContainerStarted","Data":"f999dabed223234246fe8cfc39c26d429668f5542d98658cbf04f538d6fc8fdb"}
Jul 28 17:52:11.076409 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:11.076101    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"
Jul 28 17:52:11.096448 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:11.096386    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2" podStartSLOduration=2.096363241 podStartE2EDuration="2.096363241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:52:11.094883837 +0000 UTC m=+2127.352493532" watchObservedRunningTime="2026-07-28 17:52:11.096363241 +0000 UTC m=+2127.353972931"
Jul 28 17:52:26.845336 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:26.845298    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"]
Jul 28 17:52:36.528330 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:36.528236    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm" podUID="9471a569-4746-4ed0-882e-b2fc4f485d0e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:52:41.865046 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:41.865002    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm" podUID="9471a569-4746-4ed0-882e-b2fc4f485d0e" containerName="main" containerID="cri-o://0394493a17a9e2f33531a009feb565d6f25562c817345914426a329434be3ae5" gracePeriod=45
Jul 28 17:52:42.082425 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.082384    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2" podUID="5e97b04b-b1e9-4d77-b46b-acce452b64d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:52:42.129778 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.129702    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"
Jul 28 17:52:42.227467 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.227431    2575 generic.go:356] "Generic (PLEG): container finished" podID="9471a569-4746-4ed0-882e-b2fc4f485d0e" containerID="0394493a17a9e2f33531a009feb565d6f25562c817345914426a329434be3ae5" exitCode=0
Jul 28 17:52:42.227714 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.227480    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"
Jul 28 17:52:42.227714 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.227550    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm" event={"ID":"9471a569-4746-4ed0-882e-b2fc4f485d0e","Type":"ContainerDied","Data":"0394493a17a9e2f33531a009feb565d6f25562c817345914426a329434be3ae5"}
Jul 28 17:52:42.227714 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.227611    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm" event={"ID":"9471a569-4746-4ed0-882e-b2fc4f485d0e","Type":"ContainerDied","Data":"34b647db7d98e2e61b8720e600e62deedaaaf883a3c3d008908a5d50ee5b771b"}
Jul 28 17:52:42.227714 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.227644    2575 scope.go:117] "RemoveContainer" containerID="0394493a17a9e2f33531a009feb565d6f25562c817345914426a329434be3ae5"
Jul 28 17:52:42.236810 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.236788    2575 scope.go:117] "RemoveContainer" containerID="0394493a17a9e2f33531a009feb565d6f25562c817345914426a329434be3ae5"
Jul 28 17:52:42.237130 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:52:42.237097    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0394493a17a9e2f33531a009feb565d6f25562c817345914426a329434be3ae5\": container with ID starting with 0394493a17a9e2f33531a009feb565d6f25562c817345914426a329434be3ae5 not found: ID does not exist" containerID="0394493a17a9e2f33531a009feb565d6f25562c817345914426a329434be3ae5"
Jul 28 17:52:42.237256 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.237139    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0394493a17a9e2f33531a009feb565d6f25562c817345914426a329434be3ae5"} err="failed to get container status \"0394493a17a9e2f33531a009feb565d6f25562c817345914426a329434be3ae5\": rpc error: code = NotFound desc = could not find container \"0394493a17a9e2f33531a009feb565d6f25562c817345914426a329434be3ae5\": container with ID starting with 0394493a17a9e2f33531a009feb565d6f25562c817345914426a329434be3ae5 not found: ID does not exist"
Jul 28 17:52:42.283504 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.283470    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9471a569-4746-4ed0-882e-b2fc4f485d0e-tls-certs\") pod \"9471a569-4746-4ed0-882e-b2fc4f485d0e\" (UID: \"9471a569-4746-4ed0-882e-b2fc4f485d0e\") "
Jul 28 17:52:42.283714 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.283612    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f5dh\" (UniqueName: \"kubernetes.io/projected/9471a569-4746-4ed0-882e-b2fc4f485d0e-kube-api-access-6f5dh\") pod \"9471a569-4746-4ed0-882e-b2fc4f485d0e\" (UID: \"9471a569-4746-4ed0-882e-b2fc4f485d0e\") "
Jul 28 17:52:42.285892 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.285859    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9471a569-4746-4ed0-882e-b2fc4f485d0e-kube-api-access-6f5dh" (OuterVolumeSpecName: "kube-api-access-6f5dh") pod "9471a569-4746-4ed0-882e-b2fc4f485d0e" (UID: "9471a569-4746-4ed0-882e-b2fc4f485d0e"). InnerVolumeSpecName "kube-api-access-6f5dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:52:42.285892 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.285879    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9471a569-4746-4ed0-882e-b2fc4f485d0e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9471a569-4746-4ed0-882e-b2fc4f485d0e" (UID: "9471a569-4746-4ed0-882e-b2fc4f485d0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:52:42.384490 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.384372    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f5dh\" (UniqueName: \"kubernetes.io/projected/9471a569-4746-4ed0-882e-b2fc4f485d0e-kube-api-access-6f5dh\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:52:42.384490 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.384413    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9471a569-4746-4ed0-882e-b2fc4f485d0e-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:52:42.544111 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.544076    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"]
Jul 28 17:52:42.548309 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.548274    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-5wtvm"]
Jul 28 17:52:42.729895 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.729859    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"]
Jul 28 17:52:42.731688 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:42.731650    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2" podUID="5e97b04b-b1e9-4d77-b46b-acce452b64d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:52:44.312295 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:44.312254    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9471a569-4746-4ed0-882e-b2fc4f485d0e" path="/var/lib/kubelet/pods/9471a569-4746-4ed0-882e-b2fc4f485d0e/volumes"
Jul 28 17:52:52.732728 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:52.732676    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2" podUID="5e97b04b-b1e9-4d77-b46b-acce452b64d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:52:57.751024 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:57.750954    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2" podUID="5e97b04b-b1e9-4d77-b46b-acce452b64d3" containerName="main" containerID="cri-o://e985644e880ba945851fffaa6330bd512a4c85feb7c38e00736528520df0b358" gracePeriod=45
Jul 28 17:52:58.014146 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.014067    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"
Jul 28 17:52:58.125963 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.125916    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e97b04b-b1e9-4d77-b46b-acce452b64d3-tls-certs\") pod \"5e97b04b-b1e9-4d77-b46b-acce452b64d3\" (UID: \"5e97b04b-b1e9-4d77-b46b-acce452b64d3\") "
Jul 28 17:52:58.125963 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.125970    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckgc6\" (UniqueName: \"kubernetes.io/projected/5e97b04b-b1e9-4d77-b46b-acce452b64d3-kube-api-access-ckgc6\") pod \"5e97b04b-b1e9-4d77-b46b-acce452b64d3\" (UID: \"5e97b04b-b1e9-4d77-b46b-acce452b64d3\") "
Jul 28 17:52:58.128362 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.128322    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e97b04b-b1e9-4d77-b46b-acce452b64d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e97b04b-b1e9-4d77-b46b-acce452b64d3" (UID: "5e97b04b-b1e9-4d77-b46b-acce452b64d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:52:58.128496 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.128365    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e97b04b-b1e9-4d77-b46b-acce452b64d3-kube-api-access-ckgc6" (OuterVolumeSpecName: "kube-api-access-ckgc6") pod "5e97b04b-b1e9-4d77-b46b-acce452b64d3" (UID: "5e97b04b-b1e9-4d77-b46b-acce452b64d3"). InnerVolumeSpecName "kube-api-access-ckgc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:52:58.227730 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.227685    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e97b04b-b1e9-4d77-b46b-acce452b64d3-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:52:58.227913 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.227743    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckgc6\" (UniqueName: \"kubernetes.io/projected/5e97b04b-b1e9-4d77-b46b-acce452b64d3-kube-api-access-ckgc6\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:52:58.313912 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.313817    2575 generic.go:356] "Generic (PLEG): container finished" podID="5e97b04b-b1e9-4d77-b46b-acce452b64d3" containerID="e985644e880ba945851fffaa6330bd512a4c85feb7c38e00736528520df0b358" exitCode=0
Jul 28 17:52:58.313912 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.313858    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"
Jul 28 17:52:58.313912 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.313897    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2" event={"ID":"5e97b04b-b1e9-4d77-b46b-acce452b64d3","Type":"ContainerDied","Data":"e985644e880ba945851fffaa6330bd512a4c85feb7c38e00736528520df0b358"}
Jul 28 17:52:58.314180 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.313950    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2" event={"ID":"5e97b04b-b1e9-4d77-b46b-acce452b64d3","Type":"ContainerDied","Data":"f999dabed223234246fe8cfc39c26d429668f5542d98658cbf04f538d6fc8fdb"}
Jul 28 17:52:58.314180 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.313976    2575 scope.go:117] "RemoveContainer" containerID="e985644e880ba945851fffaa6330bd512a4c85feb7c38e00736528520df0b358"
Jul 28 17:52:58.324388 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.324361    2575 scope.go:117] "RemoveContainer" containerID="e985644e880ba945851fffaa6330bd512a4c85feb7c38e00736528520df0b358"
Jul 28 17:52:58.324733 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:52:58.324707    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e985644e880ba945851fffaa6330bd512a4c85feb7c38e00736528520df0b358\": container with ID starting with e985644e880ba945851fffaa6330bd512a4c85feb7c38e00736528520df0b358 not found: ID does not exist" containerID="e985644e880ba945851fffaa6330bd512a4c85feb7c38e00736528520df0b358"
Jul 28 17:52:58.324812 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.324746    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e985644e880ba945851fffaa6330bd512a4c85feb7c38e00736528520df0b358"} err="failed to get container status \"e985644e880ba945851fffaa6330bd512a4c85feb7c38e00736528520df0b358\": rpc error: code = NotFound desc = could not find container \"e985644e880ba945851fffaa6330bd512a4c85feb7c38e00736528520df0b358\": container with ID starting with e985644e880ba945851fffaa6330bd512a4c85feb7c38e00736528520df0b358 not found: ID does not exist"
Jul 28 17:52:58.338101 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.338062    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"]
Jul 28 17:52:58.341821 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:52:58.341787    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf9srn2"]
Jul 28 17:53:00.312446 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:00.312413    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e97b04b-b1e9-4d77-b46b-acce452b64d3" path="/var/lib/kubelet/pods/5e97b04b-b1e9-4d77-b46b-acce452b64d3/volumes"
Jul 28 17:53:22.696631 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.696486    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"]
Jul 28 17:53:22.697170 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.697060    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9471a569-4746-4ed0-882e-b2fc4f485d0e" containerName="main"
Jul 28 17:53:22.697170 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.697081    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9471a569-4746-4ed0-882e-b2fc4f485d0e" containerName="main"
Jul 28 17:53:22.697170 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.697096    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e97b04b-b1e9-4d77-b46b-acce452b64d3" containerName="main"
Jul 28 17:53:22.697170 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.697104    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e97b04b-b1e9-4d77-b46b-acce452b64d3" containerName="main"
Jul 28 17:53:22.697389 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.697211    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9471a569-4746-4ed0-882e-b2fc4f485d0e" containerName="main"
Jul 28 17:53:22.697389 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.697225    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e97b04b-b1e9-4d77-b46b-acce452b64d3" containerName="main"
Jul 28 17:53:22.700495 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.700463    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.703708 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.703678    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 28 17:53:22.703708 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.703678    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-k59xv\""
Jul 28 17:53:22.703937 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.703681    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 28 17:53:22.703937 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.703681    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 28 17:53:22.710104 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.710078    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"]
Jul 28 17:53:22.756505 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.756462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572df932-c0ac-4c18-9a09-82d26b6c1cb8-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.756731 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.756563    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.756731 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.756595    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgz94\" (UniqueName: \"kubernetes.io/projected/572df932-c0ac-4c18-9a09-82d26b6c1cb8-kube-api-access-kgz94\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.756731 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.756633    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.756731 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.756676    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.756731 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.756698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.857638 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.857596    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.857815 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.857645    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.857815 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.857781    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.857947 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.857844    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572df932-c0ac-4c18-9a09-82d26b6c1cb8-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.857947 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.857933    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.858065 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.857975    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgz94\" (UniqueName: \"kubernetes.io/projected/572df932-c0ac-4c18-9a09-82d26b6c1cb8-kube-api-access-kgz94\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.858065 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.858054    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.858191 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.858095    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.858245 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.858192    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.860230 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.860202    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.860548 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.860502    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572df932-c0ac-4c18-9a09-82d26b6c1cb8-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.869154 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.869126    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgz94\" (UniqueName: \"kubernetes.io/projected/572df932-c0ac-4c18-9a09-82d26b6c1cb8-kube-api-access-kgz94\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-pqnzk\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:22.908061 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.908024    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"]
Jul 28 17:53:22.911703 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.911676    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:22.913899 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.913874    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-zp9qx\""
Jul 28 17:53:22.926732 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:22.926701    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"]
Jul 28 17:53:23.012380 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.012284    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:23.060586 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.060515    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795d8mj2\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:23.060847 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.060626    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795d8mj2\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:23.060847 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.060719    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5lg9\" (UniqueName: \"kubernetes.io/projected/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-kube-api-access-v5lg9\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795d8mj2\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:23.060847 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.060800    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795d8mj2\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:23.166739 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.164229    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"]
Jul 28 17:53:23.166739 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.164963    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795d8mj2\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:23.166739 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.165020    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795d8mj2\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:23.166739 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.165069    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795d8mj2\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:23.166739 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.165158    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5lg9\" (UniqueName: \"kubernetes.io/projected/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-kube-api-access-v5lg9\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795d8mj2\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:23.166739 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.165976    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795d8mj2\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:23.166739 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.166243    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795d8mj2\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:23.169124 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.169098    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795d8mj2\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:23.170866 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:53:23.170841    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572df932_c0ac_4c18_9a09_82d26b6c1cb8.slice/crio-51bbf85c12cd22f8d8701517255234a3993b105e440e457de0c3b22e9de8a31d WatchSource:0}: Error finding container 51bbf85c12cd22f8d8701517255234a3993b105e440e457de0c3b22e9de8a31d: Status 404 returned error can't find the container with id 51bbf85c12cd22f8d8701517255234a3993b105e440e457de0c3b22e9de8a31d
Jul 28 17:53:23.174124 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.174090    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5lg9\" (UniqueName: \"kubernetes.io/projected/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-kube-api-access-v5lg9\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795d8mj2\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:23.226271 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.226241    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:23.370026 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.369999    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"]
Jul 28 17:53:23.372722 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:53:23.372688    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9335aba_e078_48de_9ed6_f1a7a4ccb96d.slice/crio-3f0c857d5fd5259ecfb6dca2d79ea0524ffcedb05c46f37e6660f537bd7582a6 WatchSource:0}: Error finding container 3f0c857d5fd5259ecfb6dca2d79ea0524ffcedb05c46f37e6660f537bd7582a6: Status 404 returned error can't find the container with id 3f0c857d5fd5259ecfb6dca2d79ea0524ffcedb05c46f37e6660f537bd7582a6
Jul 28 17:53:23.441748 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.441702    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2" event={"ID":"a9335aba-e078-48de-9ed6-f1a7a4ccb96d","Type":"ContainerStarted","Data":"48aef7f33ebe736c234a3945199c1808a306713af69432ec77ab969c0e694a74"}
Jul 28 17:53:23.441941 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.441774    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2" event={"ID":"a9335aba-e078-48de-9ed6-f1a7a4ccb96d","Type":"ContainerStarted","Data":"3f0c857d5fd5259ecfb6dca2d79ea0524ffcedb05c46f37e6660f537bd7582a6"}
Jul 28 17:53:23.441941 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.441921    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:23.443474 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.443442    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk" event={"ID":"572df932-c0ac-4c18-9a09-82d26b6c1cb8","Type":"ContainerStarted","Data":"cf597f86be0f5bff8199e470288cca6ee1b1f7007c01c1cc0d1ab0334fa8d4c8"}
Jul 28 17:53:23.443653 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.443493    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk" event={"ID":"572df932-c0ac-4c18-9a09-82d26b6c1cb8","Type":"ContainerStarted","Data":"51bbf85c12cd22f8d8701517255234a3993b105e440e457de0c3b22e9de8a31d"}
Jul 28 17:53:23.459624 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.459567    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2" podStartSLOduration=1.4595434250000001 podStartE2EDuration="1.459543425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:53:23.458205693 +0000 UTC m=+2199.715815378" watchObservedRunningTime="2026-07-28 17:53:23.459543425 +0000 UTC m=+2199.717153119"
Jul 28 17:53:23.475136 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:23.475079    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk" podStartSLOduration=1.475061224 podStartE2EDuration="1.475061224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:53:23.474385313 +0000 UTC m=+2199.731995005" watchObservedRunningTime="2026-07-28 17:53:23.475061224 +0000 UTC m=+2199.732670913"
Jul 28 17:53:24.012981 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:24.012938    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:33.013123 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:33.013070    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:33.025675 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:33.025642    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:33.505279 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:33.505240    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:54.450656 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:54.450624    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:53:56.167411 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.167354    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"]
Jul 28 17:53:56.170391 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.170343    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"]
Jul 28 17:53:56.180325 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:53:56.180282    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:53:56.180325 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:53:56.180325 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 17:53:56.180325 ip-10-0-135-188 kubenswrapper[2575]:  > containerID="cf597f86be0f5bff8199e470288cca6ee1b1f7007c01c1cc0d1ab0334fa8d4c8" cmd=["/bin/sleep","15"]
Jul 28 17:53:56.180701 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.180344    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk" message="" err=<
Jul 28 17:53:56.180701 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:53:56.180701 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 17:53:56.180701 ip-10-0-135-188 kubenswrapper[2575]:  >
Jul 28 17:53:56.180701 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:53:56.180364    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:53:56.180701 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:53:56.180701 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 17:53:56.180701 ip-10-0-135-188 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk" podUID="572df932-c0ac-4c18-9a09-82d26b6c1cb8" containerName="main" containerID="cri-o://cf597f86be0f5bff8199e470288cca6ee1b1f7007c01c1cc0d1ab0334fa8d4c8"
Jul 28 17:53:56.180701 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.180410    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk" podUID="572df932-c0ac-4c18-9a09-82d26b6c1cb8" containerName="main" containerID="cri-o://cf597f86be0f5bff8199e470288cca6ee1b1f7007c01c1cc0d1ab0334fa8d4c8" gracePeriod=60
Jul 28 17:53:56.543624 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.543587    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:56.583471 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.583427    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-model-cache\") pod \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") "
Jul 28 17:53:56.583693 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.583477    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-home\") pod \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") "
Jul 28 17:53:56.583693 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.583553    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgz94\" (UniqueName: \"kubernetes.io/projected/572df932-c0ac-4c18-9a09-82d26b6c1cb8-kube-api-access-kgz94\") pod \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") "
Jul 28 17:53:56.583693 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.583579    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572df932-c0ac-4c18-9a09-82d26b6c1cb8-tls-certs\") pod \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") "
Jul 28 17:53:56.583693 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.583599    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-tmp-dir\") pod \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") "
Jul 28 17:53:56.583693 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.583670    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-dshm\") pod \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\" (UID: \"572df932-c0ac-4c18-9a09-82d26b6c1cb8\") "
Jul 28 17:53:56.583964 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.583787    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "572df932-c0ac-4c18-9a09-82d26b6c1cb8" (UID: "572df932-c0ac-4c18-9a09-82d26b6c1cb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:56.583964 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.583908    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "572df932-c0ac-4c18-9a09-82d26b6c1cb8" (UID: "572df932-c0ac-4c18-9a09-82d26b6c1cb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:56.583964 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.583919    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-home" (OuterVolumeSpecName: "home") pod "572df932-c0ac-4c18-9a09-82d26b6c1cb8" (UID: "572df932-c0ac-4c18-9a09-82d26b6c1cb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:56.584124 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.584025    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-tmp-dir\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:53:56.584124 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.584050    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-model-cache\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:53:56.584124 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.584065    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-home\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:53:56.586047 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.586014    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/572df932-c0ac-4c18-9a09-82d26b6c1cb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "572df932-c0ac-4c18-9a09-82d26b6c1cb8" (UID: "572df932-c0ac-4c18-9a09-82d26b6c1cb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:53:56.586204 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.586162    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-dshm" (OuterVolumeSpecName: "dshm") pod "572df932-c0ac-4c18-9a09-82d26b6c1cb8" (UID: "572df932-c0ac-4c18-9a09-82d26b6c1cb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:56.586268 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.586253    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572df932-c0ac-4c18-9a09-82d26b6c1cb8-kube-api-access-kgz94" (OuterVolumeSpecName: "kube-api-access-kgz94") pod "572df932-c0ac-4c18-9a09-82d26b6c1cb8" (UID: "572df932-c0ac-4c18-9a09-82d26b6c1cb8"). InnerVolumeSpecName "kube-api-access-kgz94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:53:56.609753 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.609716    2575 generic.go:356] "Generic (PLEG): container finished" podID="572df932-c0ac-4c18-9a09-82d26b6c1cb8" containerID="cf597f86be0f5bff8199e470288cca6ee1b1f7007c01c1cc0d1ab0334fa8d4c8" exitCode=0
Jul 28 17:53:56.609966 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.609782    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"
Jul 28 17:53:56.609966 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.609801    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk" event={"ID":"572df932-c0ac-4c18-9a09-82d26b6c1cb8","Type":"ContainerDied","Data":"cf597f86be0f5bff8199e470288cca6ee1b1f7007c01c1cc0d1ab0334fa8d4c8"}
Jul 28 17:53:56.609966 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.609853    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk" event={"ID":"572df932-c0ac-4c18-9a09-82d26b6c1cb8","Type":"ContainerDied","Data":"51bbf85c12cd22f8d8701517255234a3993b105e440e457de0c3b22e9de8a31d"}
Jul 28 17:53:56.609966 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.609875    2575 scope.go:117] "RemoveContainer" containerID="cf597f86be0f5bff8199e470288cca6ee1b1f7007c01c1cc0d1ab0334fa8d4c8"
Jul 28 17:53:56.619765 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.619745    2575 scope.go:117] "RemoveContainer" containerID="cf597f86be0f5bff8199e470288cca6ee1b1f7007c01c1cc0d1ab0334fa8d4c8"
Jul 28 17:53:56.620089 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:53:56.620068    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf597f86be0f5bff8199e470288cca6ee1b1f7007c01c1cc0d1ab0334fa8d4c8\": container with ID starting with cf597f86be0f5bff8199e470288cca6ee1b1f7007c01c1cc0d1ab0334fa8d4c8 not found: ID does not exist" containerID="cf597f86be0f5bff8199e470288cca6ee1b1f7007c01c1cc0d1ab0334fa8d4c8"
Jul 28 17:53:56.620172 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.620102    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf597f86be0f5bff8199e470288cca6ee1b1f7007c01c1cc0d1ab0334fa8d4c8"} err="failed to get container status \"cf597f86be0f5bff8199e470288cca6ee1b1f7007c01c1cc0d1ab0334fa8d4c8\": rpc error: code = NotFound desc = could not find container \"cf597f86be0f5bff8199e470288cca6ee1b1f7007c01c1cc0d1ab0334fa8d4c8\": container with ID starting with cf597f86be0f5bff8199e470288cca6ee1b1f7007c01c1cc0d1ab0334fa8d4c8 not found: ID does not exist"
Jul 28 17:53:56.633696 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.633655    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"]
Jul 28 17:53:56.637280 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.637247    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk"]
Jul 28 17:53:56.685229 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.685186    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572df932-c0ac-4c18-9a09-82d26b6c1cb8-dshm\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:53:56.685229 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.685221    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgz94\" (UniqueName: \"kubernetes.io/projected/572df932-c0ac-4c18-9a09-82d26b6c1cb8-kube-api-access-kgz94\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:53:56.685229 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:56.685232    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572df932-c0ac-4c18-9a09-82d26b6c1cb8-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:53:57.495588 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:57.495514    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-pqnzk" podUID="572df932-c0ac-4c18-9a09-82d26b6c1cb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": context deadline exceeded"
Jul 28 17:53:58.312515 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:53:58.312474    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="572df932-c0ac-4c18-9a09-82d26b6c1cb8" path="/var/lib/kubelet/pods/572df932-c0ac-4c18-9a09-82d26b6c1cb8/volumes"
Jul 28 17:54:04.450860 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:04.450812    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2" podUID="a9335aba-e078-48de-9ed6-f1a7a4ccb96d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:54:11.190442 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.190321    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2" podUID="a9335aba-e078-48de-9ed6-f1a7a4ccb96d" containerName="main" containerID="cri-o://48aef7f33ebe736c234a3945199c1808a306713af69432ec77ab969c0e694a74" gracePeriod=45
Jul 28 17:54:11.452090 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.452017    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:54:11.518843 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.518808    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-cachi2-cache\") pod \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") "
Jul 28 17:54:11.518843 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.518849    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-tokenizer-cache\") pod \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") "
Jul 28 17:54:11.519102 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.518916    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5lg9\" (UniqueName: \"kubernetes.io/projected/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-kube-api-access-v5lg9\") pod \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") "
Jul 28 17:54:11.519102 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.518944    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-tls-certs\") pod \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\" (UID: \"a9335aba-e078-48de-9ed6-f1a7a4ccb96d\") "
Jul 28 17:54:11.519182 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.519094    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "a9335aba-e078-48de-9ed6-f1a7a4ccb96d" (UID: "a9335aba-e078-48de-9ed6-f1a7a4ccb96d"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:54:11.519182 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.519153    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "a9335aba-e078-48de-9ed6-f1a7a4ccb96d" (UID: "a9335aba-e078-48de-9ed6-f1a7a4ccb96d"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:54:11.519257 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.519207    2575 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-cachi2-cache\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:54:11.521205 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.521180    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9335aba-e078-48de-9ed6-f1a7a4ccb96d" (UID: "a9335aba-e078-48de-9ed6-f1a7a4ccb96d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:54:11.521335 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.521244    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-kube-api-access-v5lg9" (OuterVolumeSpecName: "kube-api-access-v5lg9") pod "a9335aba-e078-48de-9ed6-f1a7a4ccb96d" (UID: "a9335aba-e078-48de-9ed6-f1a7a4ccb96d"). InnerVolumeSpecName "kube-api-access-v5lg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:54:11.620410 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.620357    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5lg9\" (UniqueName: \"kubernetes.io/projected/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-kube-api-access-v5lg9\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:54:11.620410 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.620399    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:54:11.620410 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.620409    2575 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9335aba-e078-48de-9ed6-f1a7a4ccb96d-tokenizer-cache\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:54:11.684705 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.684663    2575 generic.go:356] "Generic (PLEG): container finished" podID="a9335aba-e078-48de-9ed6-f1a7a4ccb96d" containerID="48aef7f33ebe736c234a3945199c1808a306713af69432ec77ab969c0e694a74" exitCode=0
Jul 28 17:54:11.684893 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.684717    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"
Jul 28 17:54:11.684893 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.684747    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2" event={"ID":"a9335aba-e078-48de-9ed6-f1a7a4ccb96d","Type":"ContainerDied","Data":"48aef7f33ebe736c234a3945199c1808a306713af69432ec77ab969c0e694a74"}
Jul 28 17:54:11.684893 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.684807    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2" event={"ID":"a9335aba-e078-48de-9ed6-f1a7a4ccb96d","Type":"ContainerDied","Data":"3f0c857d5fd5259ecfb6dca2d79ea0524ffcedb05c46f37e6660f537bd7582a6"}
Jul 28 17:54:11.684893 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.684831    2575 scope.go:117] "RemoveContainer" containerID="48aef7f33ebe736c234a3945199c1808a306713af69432ec77ab969c0e694a74"
Jul 28 17:54:11.693836 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.693809    2575 scope.go:117] "RemoveContainer" containerID="48aef7f33ebe736c234a3945199c1808a306713af69432ec77ab969c0e694a74"
Jul 28 17:54:11.694205 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:54:11.694178    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48aef7f33ebe736c234a3945199c1808a306713af69432ec77ab969c0e694a74\": container with ID starting with 48aef7f33ebe736c234a3945199c1808a306713af69432ec77ab969c0e694a74 not found: ID does not exist" containerID="48aef7f33ebe736c234a3945199c1808a306713af69432ec77ab969c0e694a74"
Jul 28 17:54:11.694261 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.694216    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48aef7f33ebe736c234a3945199c1808a306713af69432ec77ab969c0e694a74"} err="failed to get container status \"48aef7f33ebe736c234a3945199c1808a306713af69432ec77ab969c0e694a74\": rpc error: code = NotFound desc = could not find container \"48aef7f33ebe736c234a3945199c1808a306713af69432ec77ab969c0e694a74\": container with ID starting with 48aef7f33ebe736c234a3945199c1808a306713af69432ec77ab969c0e694a74 not found: ID does not exist"
Jul 28 17:54:11.709004 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.708914    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"]
Jul 28 17:54:11.713154 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:11.713116    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795d8mj2"]
Jul 28 17:54:12.312650 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:12.312614    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9335aba-e078-48de-9ed6-f1a7a4ccb96d" path="/var/lib/kubelet/pods/a9335aba-e078-48de-9ed6-f1a7a4ccb96d/volumes"
Jul 28 17:54:36.445220 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.445180    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"]
Jul 28 17:54:36.447653 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.445732    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9335aba-e078-48de-9ed6-f1a7a4ccb96d" containerName="main"
Jul 28 17:54:36.447653 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.445750    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9335aba-e078-48de-9ed6-f1a7a4ccb96d" containerName="main"
Jul 28 17:54:36.447653 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.445761    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="572df932-c0ac-4c18-9a09-82d26b6c1cb8" containerName="main"
Jul 28 17:54:36.447653 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.445766    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="572df932-c0ac-4c18-9a09-82d26b6c1cb8" containerName="main"
Jul 28 17:54:36.447653 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.445842    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9335aba-e078-48de-9ed6-f1a7a4ccb96d" containerName="main"
Jul 28 17:54:36.447653 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.445851    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="572df932-c0ac-4c18-9a09-82d26b6c1cb8" containerName="main"
Jul 28 17:54:36.449059 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.449037    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.452098 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.452067    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-wfplp\""
Jul 28 17:54:36.458498 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.458432    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-wd9ww\""
Jul 28 17:54:36.458711 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.458689    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 28 17:54:36.458816 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.458795    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 28 17:54:36.458883 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.458832    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 28 17:54:36.466027 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.465998    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"]
Jul 28 17:54:36.551273 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.551229    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e82679a6-9909-4e96-9554-44e4e239ab5d-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.551273 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.551277    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.551556 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.551354    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.551556 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.551376    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-844bz\" (UniqueName: \"kubernetes.io/projected/e82679a6-9909-4e96-9554-44e4e239ab5d-kube-api-access-844bz\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.551556 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.551445    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.551556 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.551480    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.652997 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.652952    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e82679a6-9909-4e96-9554-44e4e239ab5d-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.652997 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.652997    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.653228 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.653173    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.653228 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.653209    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-844bz\" (UniqueName: \"kubernetes.io/projected/e82679a6-9909-4e96-9554-44e4e239ab5d-kube-api-access-844bz\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.653374 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.653351    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.653438 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.653394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.653568 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.653506    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.653736 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.653715    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.653812 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.653768    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.655541 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.655506    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.655765 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.655745    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e82679a6-9909-4e96-9554-44e4e239ab5d-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.664749 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.664714    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-844bz\" (UniqueName: \"kubernetes.io/projected/e82679a6-9909-4e96-9554-44e4e239ab5d-kube-api-access-844bz\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.667655 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.667619    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"]
Jul 28 17:54:36.671450 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.671422    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"
Jul 28 17:54:36.673810 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.673781    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-sghl8\""
Jul 28 17:54:36.683253 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.683225    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"]
Jul 28 17:54:36.754765 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.754661    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fedd0af-fe13-4ebb-9de3-941e37a653bb-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r\" (UID: \"0fedd0af-fe13-4ebb-9de3-941e37a653bb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"
Jul 28 17:54:36.754765 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.754706    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcwcb\" (UniqueName: \"kubernetes.io/projected/0fedd0af-fe13-4ebb-9de3-941e37a653bb-kube-api-access-qcwcb\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r\" (UID: \"0fedd0af-fe13-4ebb-9de3-941e37a653bb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"
Jul 28 17:54:36.766650 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.766613    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:36.856267 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.856227    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fedd0af-fe13-4ebb-9de3-941e37a653bb-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r\" (UID: \"0fedd0af-fe13-4ebb-9de3-941e37a653bb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"
Jul 28 17:54:36.856427 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.856284    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcwcb\" (UniqueName: \"kubernetes.io/projected/0fedd0af-fe13-4ebb-9de3-941e37a653bb-kube-api-access-qcwcb\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r\" (UID: \"0fedd0af-fe13-4ebb-9de3-941e37a653bb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"
Jul 28 17:54:36.859308 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.859274    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fedd0af-fe13-4ebb-9de3-941e37a653bb-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r\" (UID: \"0fedd0af-fe13-4ebb-9de3-941e37a653bb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"
Jul 28 17:54:36.868189 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.868125    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcwcb\" (UniqueName: \"kubernetes.io/projected/0fedd0af-fe13-4ebb-9de3-941e37a653bb-kube-api-access-qcwcb\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r\" (UID: \"0fedd0af-fe13-4ebb-9de3-941e37a653bb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"
Jul 28 17:54:36.918620 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.918585    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"]
Jul 28 17:54:36.921348 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:54:36.921316    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82679a6_9909_4e96_9554_44e4e239ab5d.slice/crio-e9286e6fd1d0a16290b24b4de0c966bee2a9c0d3836ab4d1bc55a7546c58bfcf WatchSource:0}: Error finding container e9286e6fd1d0a16290b24b4de0c966bee2a9c0d3836ab4d1bc55a7546c58bfcf: Status 404 returned error can't find the container with id e9286e6fd1d0a16290b24b4de0c966bee2a9c0d3836ab4d1bc55a7546c58bfcf
Jul 28 17:54:36.984086 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:36.984053    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"
Jul 28 17:54:37.125361 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:37.125331    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"]
Jul 28 17:54:37.127557 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:54:37.127503    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fedd0af_fe13_4ebb_9de3_941e37a653bb.slice/crio-a679103169b1d9caa532f9a0e65cc99730bb61f34c2987cb07d79f6bdc89d709 WatchSource:0}: Error finding container a679103169b1d9caa532f9a0e65cc99730bb61f34c2987cb07d79f6bdc89d709: Status 404 returned error can't find the container with id a679103169b1d9caa532f9a0e65cc99730bb61f34c2987cb07d79f6bdc89d709
Jul 28 17:54:37.811212 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:37.811178    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r" event={"ID":"0fedd0af-fe13-4ebb-9de3-941e37a653bb","Type":"ContainerStarted","Data":"925ea91826bd053eb861a2d8bfc9273b2abf0e8dbffdf5241cc9901d6cb9ecde"}
Jul 28 17:54:37.811703 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:37.811243    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"
Jul 28 17:54:37.811703 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:37.811261    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r" event={"ID":"0fedd0af-fe13-4ebb-9de3-941e37a653bb","Type":"ContainerStarted","Data":"a679103169b1d9caa532f9a0e65cc99730bb61f34c2987cb07d79f6bdc89d709"}
Jul 28 17:54:37.812491 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:37.812466    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w" event={"ID":"e82679a6-9909-4e96-9554-44e4e239ab5d","Type":"ContainerStarted","Data":"e9286e6fd1d0a16290b24b4de0c966bee2a9c0d3836ab4d1bc55a7546c58bfcf"}
Jul 28 17:54:37.828255 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:37.828188    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r" podStartSLOduration=1.828165293 podStartE2EDuration="1.828165293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:54:37.82685963 +0000 UTC m=+2274.084469322" watchObservedRunningTime="2026-07-28 17:54:37.828165293 +0000 UTC m=+2274.085774984"
Jul 28 17:54:38.819817 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:38.819773    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w" event={"ID":"e82679a6-9909-4e96-9554-44e4e239ab5d","Type":"ContainerStarted","Data":"b3ede9742ac1be3f604eb8e130458d82c0bf7de015d543114c0c63ef742e074c"}
Jul 28 17:54:38.820293 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:38.820127    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:39.828214 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:39.828175    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w" event={"ID":"e82679a6-9909-4e96-9554-44e4e239ab5d","Type":"ContainerStarted","Data":"a79a5adb4f53778832ff0970b2a4f7cbb64fb1772cc19dedb85b12ef0a034be7"}
Jul 28 17:54:39.849122 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:39.849065    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w" podStartSLOduration=2.339987227 podStartE2EDuration="3.849044394s" totalImagesPullingTime="1.509057167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:54:39.848005719 +0000 UTC m=+2276.105615410" watchObservedRunningTime="2026-07-28 17:54:39.849044394 +0000 UTC m=+2276.106654085"
Jul 28 17:54:40.767717 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:40.767670    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:46.767255 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:46.767205    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:46.779843 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:46.779809    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:46.881444 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:46.881410    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:54:57.891321 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:54:57.891284    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:55:08.821813 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:08.821774    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"
Jul 28 17:55:10.690565 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:10.690504    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"]
Jul 28 17:55:10.693849 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:10.693819    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"]
Jul 28 17:55:10.702896 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:55:10.702850    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:55:10.702896 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:55:10.702896 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 17:55:10.702896 ip-10-0-135-188 kubenswrapper[2575]:  > containerID="a79a5adb4f53778832ff0970b2a4f7cbb64fb1772cc19dedb85b12ef0a034be7" cmd=["/bin/sleep","15"]
Jul 28 17:55:10.703180 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:10.702912    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w" message="" err=<
Jul 28 17:55:10.703180 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:55:10.703180 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 17:55:10.703180 ip-10-0-135-188 kubenswrapper[2575]:  >
Jul 28 17:55:10.703180 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:55:10.702926    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:55:10.703180 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:55:10.703180 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 17:55:10.703180 ip-10-0-135-188 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w" podUID="e82679a6-9909-4e96-9554-44e4e239ab5d" containerName="main" containerID="cri-o://a79a5adb4f53778832ff0970b2a4f7cbb64fb1772cc19dedb85b12ef0a034be7"
Jul 28 17:55:10.703180 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:10.702966    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w" podUID="e82679a6-9909-4e96-9554-44e4e239ab5d" containerName="main" containerID="cri-o://a79a5adb4f53778832ff0970b2a4f7cbb64fb1772cc19dedb85b12ef0a034be7" gracePeriod=60
Jul 28 17:55:10.870899 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:10.870854    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w" podUID="e82679a6-9909-4e96-9554-44e4e239ab5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 28 17:55:10.919511 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:10.919443    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w" podUID="e82679a6-9909-4e96-9554-44e4e239ab5d" containerName="llm-d-routing-sidecar" containerID="cri-o://b3ede9742ac1be3f604eb8e130458d82c0bf7de015d543114c0c63ef742e074c" gracePeriod=60
Jul 28 17:55:10.989362 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:10.989245    2575 generic.go:356] "Generic (PLEG): container finished" podID="e82679a6-9909-4e96-9554-44e4e239ab5d" containerID="a79a5adb4f53778832ff0970b2a4f7cbb64fb1772cc19dedb85b12ef0a034be7" exitCode=0
Jul 28 17:55:10.989362 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:10.989319    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w" event={"ID":"e82679a6-9909-4e96-9554-44e4e239ab5d","Type":"ContainerDied","Data":"a79a5adb4f53778832ff0970b2a4f7cbb64fb1772cc19dedb85b12ef0a034be7"}
Jul 28 17:55:11.183205 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.183179    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:55:11.271852 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.271748    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-home\") pod \"e82679a6-9909-4e96-9554-44e4e239ab5d\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") "
Jul 28 17:55:11.271852 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.271839    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-model-cache\") pod \"e82679a6-9909-4e96-9554-44e4e239ab5d\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") "
Jul 28 17:55:11.272098 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.271864    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-tmp-dir\") pod \"e82679a6-9909-4e96-9554-44e4e239ab5d\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") "
Jul 28 17:55:11.272098 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.271916    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e82679a6-9909-4e96-9554-44e4e239ab5d-tls-certs\") pod \"e82679a6-9909-4e96-9554-44e4e239ab5d\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") "
Jul 28 17:55:11.272098 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.271942    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-844bz\" (UniqueName: \"kubernetes.io/projected/e82679a6-9909-4e96-9554-44e4e239ab5d-kube-api-access-844bz\") pod \"e82679a6-9909-4e96-9554-44e4e239ab5d\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") "
Jul 28 17:55:11.272098 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.271970    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-dshm\") pod \"e82679a6-9909-4e96-9554-44e4e239ab5d\" (UID: \"e82679a6-9909-4e96-9554-44e4e239ab5d\") "
Jul 28 17:55:11.272098 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.272064    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-home" (OuterVolumeSpecName: "home") pod "e82679a6-9909-4e96-9554-44e4e239ab5d" (UID: "e82679a6-9909-4e96-9554-44e4e239ab5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:55:11.272291 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.272160    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e82679a6-9909-4e96-9554-44e4e239ab5d" (UID: "e82679a6-9909-4e96-9554-44e4e239ab5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:55:11.272291 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.272273    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e82679a6-9909-4e96-9554-44e4e239ab5d" (UID: "e82679a6-9909-4e96-9554-44e4e239ab5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:55:11.272396 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.272334    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-home\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:55:11.272396 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.272354    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-model-cache\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:55:11.274323 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.274297    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-dshm" (OuterVolumeSpecName: "dshm") pod "e82679a6-9909-4e96-9554-44e4e239ab5d" (UID: "e82679a6-9909-4e96-9554-44e4e239ab5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:55:11.274444 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.274344    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e82679a6-9909-4e96-9554-44e4e239ab5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e82679a6-9909-4e96-9554-44e4e239ab5d" (UID: "e82679a6-9909-4e96-9554-44e4e239ab5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:55:11.274444 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.274407    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82679a6-9909-4e96-9554-44e4e239ab5d-kube-api-access-844bz" (OuterVolumeSpecName: "kube-api-access-844bz") pod "e82679a6-9909-4e96-9554-44e4e239ab5d" (UID: "e82679a6-9909-4e96-9554-44e4e239ab5d"). InnerVolumeSpecName "kube-api-access-844bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:55:11.373245 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.373200    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-tmp-dir\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:55:11.373245 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.373237    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e82679a6-9909-4e96-9554-44e4e239ab5d-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:55:11.373245 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.373247    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-844bz\" (UniqueName: \"kubernetes.io/projected/e82679a6-9909-4e96-9554-44e4e239ab5d-kube-api-access-844bz\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:55:11.373245 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:11.373257    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e82679a6-9909-4e96-9554-44e4e239ab5d-dshm\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:55:12.002682 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:12.002644    2575 generic.go:356] "Generic (PLEG): container finished" podID="e82679a6-9909-4e96-9554-44e4e239ab5d" containerID="b3ede9742ac1be3f604eb8e130458d82c0bf7de015d543114c0c63ef742e074c" exitCode=0
Jul 28 17:55:12.003166 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:12.002716    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"
Jul 28 17:55:12.003166 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:12.002745    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w" event={"ID":"e82679a6-9909-4e96-9554-44e4e239ab5d","Type":"ContainerDied","Data":"b3ede9742ac1be3f604eb8e130458d82c0bf7de015d543114c0c63ef742e074c"}
Jul 28 17:55:12.003166 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:12.002848    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w" event={"ID":"e82679a6-9909-4e96-9554-44e4e239ab5d","Type":"ContainerDied","Data":"e9286e6fd1d0a16290b24b4de0c966bee2a9c0d3836ab4d1bc55a7546c58bfcf"}
Jul 28 17:55:12.003166 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:12.002880    2575 scope.go:117] "RemoveContainer" containerID="a79a5adb4f53778832ff0970b2a4f7cbb64fb1772cc19dedb85b12ef0a034be7"
Jul 28 17:55:12.013028 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:12.013010    2575 scope.go:117] "RemoveContainer" containerID="b3ede9742ac1be3f604eb8e130458d82c0bf7de015d543114c0c63ef742e074c"
Jul 28 17:55:12.021572 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:12.021549    2575 scope.go:117] "RemoveContainer" containerID="a79a5adb4f53778832ff0970b2a4f7cbb64fb1772cc19dedb85b12ef0a034be7"
Jul 28 17:55:12.021890 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:55:12.021852    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79a5adb4f53778832ff0970b2a4f7cbb64fb1772cc19dedb85b12ef0a034be7\": container with ID starting with a79a5adb4f53778832ff0970b2a4f7cbb64fb1772cc19dedb85b12ef0a034be7 not found: ID does not exist" containerID="a79a5adb4f53778832ff0970b2a4f7cbb64fb1772cc19dedb85b12ef0a034be7"
Jul 28 17:55:12.021953 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:12.021918    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79a5adb4f53778832ff0970b2a4f7cbb64fb1772cc19dedb85b12ef0a034be7"} err="failed to get container status \"a79a5adb4f53778832ff0970b2a4f7cbb64fb1772cc19dedb85b12ef0a034be7\": rpc error: code = NotFound desc = could not find container \"a79a5adb4f53778832ff0970b2a4f7cbb64fb1772cc19dedb85b12ef0a034be7\": container with ID starting with a79a5adb4f53778832ff0970b2a4f7cbb64fb1772cc19dedb85b12ef0a034be7 not found: ID does not exist"
Jul 28 17:55:12.021997 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:12.021962    2575 scope.go:117] "RemoveContainer" containerID="b3ede9742ac1be3f604eb8e130458d82c0bf7de015d543114c0c63ef742e074c"
Jul 28 17:55:12.022298 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:55:12.022268    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ede9742ac1be3f604eb8e130458d82c0bf7de015d543114c0c63ef742e074c\": container with ID starting with b3ede9742ac1be3f604eb8e130458d82c0bf7de015d543114c0c63ef742e074c not found: ID does not exist" containerID="b3ede9742ac1be3f604eb8e130458d82c0bf7de015d543114c0c63ef742e074c"
Jul 28 17:55:12.022375 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:12.022305    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ede9742ac1be3f604eb8e130458d82c0bf7de015d543114c0c63ef742e074c"} err="failed to get container status \"b3ede9742ac1be3f604eb8e130458d82c0bf7de015d543114c0c63ef742e074c\": rpc error: code = NotFound desc = could not find container \"b3ede9742ac1be3f604eb8e130458d82c0bf7de015d543114c0c63ef742e074c\": container with ID starting with b3ede9742ac1be3f604eb8e130458d82c0bf7de015d543114c0c63ef742e074c not found: ID does not exist"
Jul 28 17:55:12.031734 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:12.031697    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"]
Jul 28 17:55:12.037321 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:12.037287    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-kp82w"]
Jul 28 17:55:12.313923 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:12.313828    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e82679a6-9909-4e96-9554-44e4e239ab5d" path="/var/lib/kubelet/pods/e82679a6-9909-4e96-9554-44e4e239ab5d/volumes"
Jul 28 17:55:18.821449 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:18.821404    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r" podUID="0fedd0af-fe13-4ebb-9de3-941e37a653bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:55:25.711700 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:25.711650    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r" podUID="0fedd0af-fe13-4ebb-9de3-941e37a653bb" containerName="main" containerID="cri-o://925ea91826bd053eb861a2d8bfc9273b2abf0e8dbffdf5241cc9901d6cb9ecde" gracePeriod=45
Jul 28 17:55:25.970375 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:25.970303    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"
Jul 28 17:55:26.007582 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.007514    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fedd0af-fe13-4ebb-9de3-941e37a653bb-tls-certs\") pod \"0fedd0af-fe13-4ebb-9de3-941e37a653bb\" (UID: \"0fedd0af-fe13-4ebb-9de3-941e37a653bb\") "
Jul 28 17:55:26.007781 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.007628    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcwcb\" (UniqueName: \"kubernetes.io/projected/0fedd0af-fe13-4ebb-9de3-941e37a653bb-kube-api-access-qcwcb\") pod \"0fedd0af-fe13-4ebb-9de3-941e37a653bb\" (UID: \"0fedd0af-fe13-4ebb-9de3-941e37a653bb\") "
Jul 28 17:55:26.009717 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.009686    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fedd0af-fe13-4ebb-9de3-941e37a653bb-kube-api-access-qcwcb" (OuterVolumeSpecName: "kube-api-access-qcwcb") pod "0fedd0af-fe13-4ebb-9de3-941e37a653bb" (UID: "0fedd0af-fe13-4ebb-9de3-941e37a653bb"). InnerVolumeSpecName "kube-api-access-qcwcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:55:26.009845 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.009783    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fedd0af-fe13-4ebb-9de3-941e37a653bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fedd0af-fe13-4ebb-9de3-941e37a653bb" (UID: "0fedd0af-fe13-4ebb-9de3-941e37a653bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:55:26.072885 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.072846    2575 generic.go:356] "Generic (PLEG): container finished" podID="0fedd0af-fe13-4ebb-9de3-941e37a653bb" containerID="925ea91826bd053eb861a2d8bfc9273b2abf0e8dbffdf5241cc9901d6cb9ecde" exitCode=0
Jul 28 17:55:26.073065 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.072897    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"
Jul 28 17:55:26.073065 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.072935    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r" event={"ID":"0fedd0af-fe13-4ebb-9de3-941e37a653bb","Type":"ContainerDied","Data":"925ea91826bd053eb861a2d8bfc9273b2abf0e8dbffdf5241cc9901d6cb9ecde"}
Jul 28 17:55:26.073065 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.073002    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r" event={"ID":"0fedd0af-fe13-4ebb-9de3-941e37a653bb","Type":"ContainerDied","Data":"a679103169b1d9caa532f9a0e65cc99730bb61f34c2987cb07d79f6bdc89d709"}
Jul 28 17:55:26.073065 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.073032    2575 scope.go:117] "RemoveContainer" containerID="925ea91826bd053eb861a2d8bfc9273b2abf0e8dbffdf5241cc9901d6cb9ecde"
Jul 28 17:55:26.084799 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.084771    2575 scope.go:117] "RemoveContainer" containerID="925ea91826bd053eb861a2d8bfc9273b2abf0e8dbffdf5241cc9901d6cb9ecde"
Jul 28 17:55:26.085111 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:55:26.085087    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925ea91826bd053eb861a2d8bfc9273b2abf0e8dbffdf5241cc9901d6cb9ecde\": container with ID starting with 925ea91826bd053eb861a2d8bfc9273b2abf0e8dbffdf5241cc9901d6cb9ecde not found: ID does not exist" containerID="925ea91826bd053eb861a2d8bfc9273b2abf0e8dbffdf5241cc9901d6cb9ecde"
Jul 28 17:55:26.085184 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.085118    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925ea91826bd053eb861a2d8bfc9273b2abf0e8dbffdf5241cc9901d6cb9ecde"} err="failed to get container status \"925ea91826bd053eb861a2d8bfc9273b2abf0e8dbffdf5241cc9901d6cb9ecde\": rpc error: code = NotFound desc = could not find container \"925ea91826bd053eb861a2d8bfc9273b2abf0e8dbffdf5241cc9901d6cb9ecde\": container with ID starting with 925ea91826bd053eb861a2d8bfc9273b2abf0e8dbffdf5241cc9901d6cb9ecde not found: ID does not exist"
Jul 28 17:55:26.104538 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.104486    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"]
Jul 28 17:55:26.109076 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.109039    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcwcb\" (UniqueName: \"kubernetes.io/projected/0fedd0af-fe13-4ebb-9de3-941e37a653bb-kube-api-access-qcwcb\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:55:26.109076 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.109072    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fedd0af-fe13-4ebb-9de3-941e37a653bb-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:55:26.114592 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.114563    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f965l9r"]
Jul 28 17:55:26.313234 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:26.313144    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fedd0af-fe13-4ebb-9de3-941e37a653bb" path="/var/lib/kubelet/pods/0fedd0af-fe13-4ebb-9de3-941e37a653bb/volumes"
Jul 28 17:55:50.668833 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.668792    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"]
Jul 28 17:55:50.669269 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.669144    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e82679a6-9909-4e96-9554-44e4e239ab5d" containerName="llm-d-routing-sidecar"
Jul 28 17:55:50.669269 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.669155    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e82679a6-9909-4e96-9554-44e4e239ab5d" containerName="llm-d-routing-sidecar"
Jul 28 17:55:50.669269 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.669170    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fedd0af-fe13-4ebb-9de3-941e37a653bb" containerName="main"
Jul 28 17:55:50.669269 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.669176    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fedd0af-fe13-4ebb-9de3-941e37a653bb" containerName="main"
Jul 28 17:55:50.669269 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.669188    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e82679a6-9909-4e96-9554-44e4e239ab5d" containerName="main"
Jul 28 17:55:50.669269 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.669194    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e82679a6-9909-4e96-9554-44e4e239ab5d" containerName="main"
Jul 28 17:55:50.669269 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.669255    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fedd0af-fe13-4ebb-9de3-941e37a653bb" containerName="main"
Jul 28 17:55:50.669269 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.669262    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e82679a6-9909-4e96-9554-44e4e239ab5d" containerName="llm-d-routing-sidecar"
Jul 28 17:55:50.669269 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.669271    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e82679a6-9909-4e96-9554-44e4e239ab5d" containerName="main"
Jul 28 17:55:50.671530 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.671497    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.676165 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.676130    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-tkvq9\""
Jul 28 17:55:50.676337 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.676140    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 28 17:55:50.676761 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.676738    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-7clcb\""
Jul 28 17:55:50.676761 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.676756    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 28 17:55:50.676978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.676776    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 28 17:55:50.693271 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.693235    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"]
Jul 28 17:55:50.832756 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.832708    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fccn2\" (UniqueName: \"kubernetes.io/projected/7d9adf4e-d6df-4481-84fd-515694f35fb4-kube-api-access-fccn2\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.832756 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.832756    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.833036 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.832885    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.833036 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.832929    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.833036 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.833003    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.833199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.833050    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9adf4e-d6df-4481-84fd-515694f35fb4-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.926381 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.926297    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"]
Jul 28 17:55:50.928960 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.928937    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"
Jul 28 17:55:50.931942 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.931921    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-h9ttn\""
Jul 28 17:55:50.933516 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.933493    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.933603 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.933564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.933646 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.933602    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.933646 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.933630    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9adf4e-d6df-4481-84fd-515694f35fb4-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.933748 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.933659    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fccn2\" (UniqueName: \"kubernetes.io/projected/7d9adf4e-d6df-4481-84fd-515694f35fb4-kube-api-access-fccn2\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.933804 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.933744    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.933935 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.933913    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.934044 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.934000    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.934142 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.934101    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.936086 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.936057    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.936375 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.936273    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9adf4e-d6df-4481-84fd-515694f35fb4-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.942385 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.942356    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"]
Jul 28 17:55:50.947846 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.947812    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fccn2\" (UniqueName: \"kubernetes.io/projected/7d9adf4e-d6df-4481-84fd-515694f35fb4-kube-api-access-fccn2\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:50.983248 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:50.983203    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:51.035231 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:51.034651    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d6c714-235e-47f5-8bfe-e4440d02cd35-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn\" (UID: \"38d6c714-235e-47f5-8bfe-e4440d02cd35\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"
Jul 28 17:55:51.035231 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:51.034758    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4nps\" (UniqueName: \"kubernetes.io/projected/38d6c714-235e-47f5-8bfe-e4440d02cd35-kube-api-access-z4nps\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn\" (UID: \"38d6c714-235e-47f5-8bfe-e4440d02cd35\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"
Jul 28 17:55:51.127380 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:51.127343    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"]
Jul 28 17:55:51.130644 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:55:51.130606    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9adf4e_d6df_4481_84fd_515694f35fb4.slice/crio-645af713f10db0851943e917904efd1e830782304b76b1faa446a581a55c1815 WatchSource:0}: Error finding container 645af713f10db0851943e917904efd1e830782304b76b1faa446a581a55c1815: Status 404 returned error can't find the container with id 645af713f10db0851943e917904efd1e830782304b76b1faa446a581a55c1815
Jul 28 17:55:51.136135 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:51.136105    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d6c714-235e-47f5-8bfe-e4440d02cd35-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn\" (UID: \"38d6c714-235e-47f5-8bfe-e4440d02cd35\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"
Jul 28 17:55:51.136251 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:51.136208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4nps\" (UniqueName: \"kubernetes.io/projected/38d6c714-235e-47f5-8bfe-e4440d02cd35-kube-api-access-z4nps\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn\" (UID: \"38d6c714-235e-47f5-8bfe-e4440d02cd35\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"
Jul 28 17:55:51.139344 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:51.139314    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d6c714-235e-47f5-8bfe-e4440d02cd35-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn\" (UID: \"38d6c714-235e-47f5-8bfe-e4440d02cd35\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"
Jul 28 17:55:51.146317 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:51.144550    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4nps\" (UniqueName: \"kubernetes.io/projected/38d6c714-235e-47f5-8bfe-e4440d02cd35-kube-api-access-z4nps\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn\" (UID: \"38d6c714-235e-47f5-8bfe-e4440d02cd35\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"
Jul 28 17:55:51.194968 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:51.194932    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97" event={"ID":"7d9adf4e-d6df-4481-84fd-515694f35fb4","Type":"ContainerStarted","Data":"645af713f10db0851943e917904efd1e830782304b76b1faa446a581a55c1815"}
Jul 28 17:55:51.250889 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:51.250848    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"
Jul 28 17:55:51.394728 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:51.394699    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"]
Jul 28 17:55:51.396929 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:55:51.396888    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d6c714_235e_47f5_8bfe_e4440d02cd35.slice/crio-18a8c77ff4cd2ea601fab709825ee379c712a176fe2fe2cc45f2d72e4954bae1 WatchSource:0}: Error finding container 18a8c77ff4cd2ea601fab709825ee379c712a176fe2fe2cc45f2d72e4954bae1: Status 404 returned error can't find the container with id 18a8c77ff4cd2ea601fab709825ee379c712a176fe2fe2cc45f2d72e4954bae1
Jul 28 17:55:52.202227 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:52.202187    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97" event={"ID":"7d9adf4e-d6df-4481-84fd-515694f35fb4","Type":"ContainerStarted","Data":"ef4cab7d595381682e7cfc2eb75b971c9bae6c54c60905d0cee92e8877f212b1"}
Jul 28 17:55:52.202713 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:52.202307    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:55:52.203876 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:52.203853    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn" event={"ID":"38d6c714-235e-47f5-8bfe-e4440d02cd35","Type":"ContainerStarted","Data":"404e07f6337805ed03c6a94e7d37d32b806f5272a6e3e6b737518e669fe69cc4"}
Jul 28 17:55:52.203995 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:52.203896    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn" event={"ID":"38d6c714-235e-47f5-8bfe-e4440d02cd35","Type":"ContainerStarted","Data":"18a8c77ff4cd2ea601fab709825ee379c712a176fe2fe2cc45f2d72e4954bae1"}
Jul 28 17:55:52.203995 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:52.203936    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"
Jul 28 17:55:52.247640 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:52.247581    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn" podStartSLOduration=2.247557424 podStartE2EDuration="2.247557424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:55:52.246888923 +0000 UTC m=+2348.504498614" watchObservedRunningTime="2026-07-28 17:55:52.247557424 +0000 UTC m=+2348.505167113"
Jul 28 17:55:53.210123 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:53.210085    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97" event={"ID":"7d9adf4e-d6df-4481-84fd-515694f35fb4","Type":"ContainerStarted","Data":"d98404733da1f6ad1661a05e376dace047097fb14311eaaf17c9ec9b55bb968c"}
Jul 28 17:55:53.239895 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:53.239825    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97" podStartSLOduration=3.239806905 podStartE2EDuration="3.239806905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:55:53.235963346 +0000 UTC m=+2349.493573035" watchObservedRunningTime="2026-07-28 17:55:53.239806905 +0000 UTC m=+2349.497416602"
Jul 28 17:55:53.984231 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:55:53.984182    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:56:00.983620 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:00.983570    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:56:00.996430 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:00.996394    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:56:01.260562 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:01.260449    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:56:01.261721 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:01.261697    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:56:23.212412 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:23.212381    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"
Jul 28 17:56:24.985138 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:24.985092    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"]
Jul 28 17:56:24.996684 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:24.996651    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"]
Jul 28 17:56:25.006070 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:56:25.006028    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:56:25.006070 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:56:25.006070 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 17:56:25.006070 ip-10-0-135-188 kubenswrapper[2575]:  > containerID="d98404733da1f6ad1661a05e376dace047097fb14311eaaf17c9ec9b55bb968c" cmd=["/bin/sleep","15"]
Jul 28 17:56:25.006339 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.006087    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97" message="" err=<
Jul 28 17:56:25.006339 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:56:25.006339 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 17:56:25.006339 ip-10-0-135-188 kubenswrapper[2575]:  >
Jul 28 17:56:25.006339 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:56:25.006101    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:56:25.006339 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:56:25.006339 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 17:56:25.006339 ip-10-0-135-188 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97" podUID="7d9adf4e-d6df-4481-84fd-515694f35fb4" containerName="main" containerID="cri-o://d98404733da1f6ad1661a05e376dace047097fb14311eaaf17c9ec9b55bb968c"
Jul 28 17:56:25.006339 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.006171    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97" podUID="7d9adf4e-d6df-4481-84fd-515694f35fb4" containerName="main" containerID="cri-o://d98404733da1f6ad1661a05e376dace047097fb14311eaaf17c9ec9b55bb968c" gracePeriod=60
Jul 28 17:56:25.222978 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.222924    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97" podUID="7d9adf4e-d6df-4481-84fd-515694f35fb4" containerName="llm-d-routing-sidecar" containerID="cri-o://ef4cab7d595381682e7cfc2eb75b971c9bae6c54c60905d0cee92e8877f212b1" gracePeriod=60
Jul 28 17:56:25.249561 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.249506    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97" podUID="7d9adf4e-d6df-4481-84fd-515694f35fb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 28 17:56:25.372866 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.372826    2575 generic.go:356] "Generic (PLEG): container finished" podID="7d9adf4e-d6df-4481-84fd-515694f35fb4" containerID="d98404733da1f6ad1661a05e376dace047097fb14311eaaf17c9ec9b55bb968c" exitCode=0
Jul 28 17:56:25.372866 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.372852    2575 generic.go:356] "Generic (PLEG): container finished" podID="7d9adf4e-d6df-4481-84fd-515694f35fb4" containerID="ef4cab7d595381682e7cfc2eb75b971c9bae6c54c60905d0cee92e8877f212b1" exitCode=0
Jul 28 17:56:25.373077 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.372893    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97" event={"ID":"7d9adf4e-d6df-4481-84fd-515694f35fb4","Type":"ContainerDied","Data":"d98404733da1f6ad1661a05e376dace047097fb14311eaaf17c9ec9b55bb968c"}
Jul 28 17:56:25.373077 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.372963    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97" event={"ID":"7d9adf4e-d6df-4481-84fd-515694f35fb4","Type":"ContainerDied","Data":"ef4cab7d595381682e7cfc2eb75b971c9bae6c54c60905d0cee92e8877f212b1"}
Jul 28 17:56:25.496945 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.496872    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:56:25.548072 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.548040    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-model-cache\") pod \"7d9adf4e-d6df-4481-84fd-515694f35fb4\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") "
Jul 28 17:56:25.548072 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.548078    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-home\") pod \"7d9adf4e-d6df-4481-84fd-515694f35fb4\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") "
Jul 28 17:56:25.548332 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.548108    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9adf4e-d6df-4481-84fd-515694f35fb4-tls-certs\") pod \"7d9adf4e-d6df-4481-84fd-515694f35fb4\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") "
Jul 28 17:56:25.548332 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.548164    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fccn2\" (UniqueName: \"kubernetes.io/projected/7d9adf4e-d6df-4481-84fd-515694f35fb4-kube-api-access-fccn2\") pod \"7d9adf4e-d6df-4481-84fd-515694f35fb4\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") "
Jul 28 17:56:25.548332 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.548230    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-tmp-dir\") pod \"7d9adf4e-d6df-4481-84fd-515694f35fb4\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") "
Jul 28 17:56:25.548332 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.548262    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-dshm\") pod \"7d9adf4e-d6df-4481-84fd-515694f35fb4\" (UID: \"7d9adf4e-d6df-4481-84fd-515694f35fb4\") "
Jul 28 17:56:25.548555 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.548380    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d9adf4e-d6df-4481-84fd-515694f35fb4" (UID: "7d9adf4e-d6df-4481-84fd-515694f35fb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:25.548555 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.548400    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-home" (OuterVolumeSpecName: "home") pod "7d9adf4e-d6df-4481-84fd-515694f35fb4" (UID: "7d9adf4e-d6df-4481-84fd-515694f35fb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:25.548555 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.548533    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-model-cache\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:56:25.548555 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.548553    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-home\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:56:25.548720 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.548630    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d9adf4e-d6df-4481-84fd-515694f35fb4" (UID: "7d9adf4e-d6df-4481-84fd-515694f35fb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:25.550605 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.550578    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-dshm" (OuterVolumeSpecName: "dshm") pod "7d9adf4e-d6df-4481-84fd-515694f35fb4" (UID: "7d9adf4e-d6df-4481-84fd-515694f35fb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:25.550825 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.550800    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d9adf4e-d6df-4481-84fd-515694f35fb4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d9adf4e-d6df-4481-84fd-515694f35fb4" (UID: "7d9adf4e-d6df-4481-84fd-515694f35fb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:56:25.550923 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.550801    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9adf4e-d6df-4481-84fd-515694f35fb4-kube-api-access-fccn2" (OuterVolumeSpecName: "kube-api-access-fccn2") pod "7d9adf4e-d6df-4481-84fd-515694f35fb4" (UID: "7d9adf4e-d6df-4481-84fd-515694f35fb4"). InnerVolumeSpecName "kube-api-access-fccn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:56:25.649969 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.649923    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-tmp-dir\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:56:25.649969 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.649966    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9adf4e-d6df-4481-84fd-515694f35fb4-dshm\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:56:25.649969 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.649976    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9adf4e-d6df-4481-84fd-515694f35fb4-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:56:25.650278 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:25.649985    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fccn2\" (UniqueName: \"kubernetes.io/projected/7d9adf4e-d6df-4481-84fd-515694f35fb4-kube-api-access-fccn2\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:56:26.380080 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:26.380045    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"
Jul 28 17:56:26.380080 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:26.380064    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97" event={"ID":"7d9adf4e-d6df-4481-84fd-515694f35fb4","Type":"ContainerDied","Data":"645af713f10db0851943e917904efd1e830782304b76b1faa446a581a55c1815"}
Jul 28 17:56:26.380581 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:26.380138    2575 scope.go:117] "RemoveContainer" containerID="d98404733da1f6ad1661a05e376dace047097fb14311eaaf17c9ec9b55bb968c"
Jul 28 17:56:26.389721 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:26.389593    2575 scope.go:117] "RemoveContainer" containerID="ef4cab7d595381682e7cfc2eb75b971c9bae6c54c60905d0cee92e8877f212b1"
Jul 28 17:56:26.420600 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:26.420560    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"]
Jul 28 17:56:26.424715 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:26.424683    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-xfc97"]
Jul 28 17:56:28.312342 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:28.312302    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d9adf4e-d6df-4481-84fd-515694f35fb4" path="/var/lib/kubelet/pods/7d9adf4e-d6df-4481-84fd-515694f35fb4/volumes"
Jul 28 17:56:33.212144 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:33.212100    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn" podUID="38d6c714-235e-47f5-8bfe-e4440d02cd35" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:56:40.005753 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.005705    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn" podUID="38d6c714-235e-47f5-8bfe-e4440d02cd35" containerName="main" containerID="cri-o://404e07f6337805ed03c6a94e7d37d32b806f5272a6e3e6b737518e669fe69cc4" gracePeriod=45
Jul 28 17:56:40.271475 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.271441    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"
Jul 28 17:56:40.384395 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.384348    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4nps\" (UniqueName: \"kubernetes.io/projected/38d6c714-235e-47f5-8bfe-e4440d02cd35-kube-api-access-z4nps\") pod \"38d6c714-235e-47f5-8bfe-e4440d02cd35\" (UID: \"38d6c714-235e-47f5-8bfe-e4440d02cd35\") "
Jul 28 17:56:40.384395 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.384400    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d6c714-235e-47f5-8bfe-e4440d02cd35-tls-certs\") pod \"38d6c714-235e-47f5-8bfe-e4440d02cd35\" (UID: \"38d6c714-235e-47f5-8bfe-e4440d02cd35\") "
Jul 28 17:56:40.387181 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.387152    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38d6c714-235e-47f5-8bfe-e4440d02cd35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38d6c714-235e-47f5-8bfe-e4440d02cd35" (UID: "38d6c714-235e-47f5-8bfe-e4440d02cd35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:56:40.387322 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.387175    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d6c714-235e-47f5-8bfe-e4440d02cd35-kube-api-access-z4nps" (OuterVolumeSpecName: "kube-api-access-z4nps") pod "38d6c714-235e-47f5-8bfe-e4440d02cd35" (UID: "38d6c714-235e-47f5-8bfe-e4440d02cd35"). InnerVolumeSpecName "kube-api-access-z4nps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:56:40.449342 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.449292    2575 generic.go:356] "Generic (PLEG): container finished" podID="38d6c714-235e-47f5-8bfe-e4440d02cd35" containerID="404e07f6337805ed03c6a94e7d37d32b806f5272a6e3e6b737518e669fe69cc4" exitCode=0
Jul 28 17:56:40.449580 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.449342    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"
Jul 28 17:56:40.449580 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.449354    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn" event={"ID":"38d6c714-235e-47f5-8bfe-e4440d02cd35","Type":"ContainerDied","Data":"404e07f6337805ed03c6a94e7d37d32b806f5272a6e3e6b737518e669fe69cc4"}
Jul 28 17:56:40.449580 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.449403    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn" event={"ID":"38d6c714-235e-47f5-8bfe-e4440d02cd35","Type":"ContainerDied","Data":"18a8c77ff4cd2ea601fab709825ee379c712a176fe2fe2cc45f2d72e4954bae1"}
Jul 28 17:56:40.449580 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.449427    2575 scope.go:117] "RemoveContainer" containerID="404e07f6337805ed03c6a94e7d37d32b806f5272a6e3e6b737518e669fe69cc4"
Jul 28 17:56:40.459113 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.459094    2575 scope.go:117] "RemoveContainer" containerID="404e07f6337805ed03c6a94e7d37d32b806f5272a6e3e6b737518e669fe69cc4"
Jul 28 17:56:40.459400 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:56:40.459378    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"404e07f6337805ed03c6a94e7d37d32b806f5272a6e3e6b737518e669fe69cc4\": container with ID starting with 404e07f6337805ed03c6a94e7d37d32b806f5272a6e3e6b737518e669fe69cc4 not found: ID does not exist" containerID="404e07f6337805ed03c6a94e7d37d32b806f5272a6e3e6b737518e669fe69cc4"
Jul 28 17:56:40.459471 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.459407    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404e07f6337805ed03c6a94e7d37d32b806f5272a6e3e6b737518e669fe69cc4"} err="failed to get container status \"404e07f6337805ed03c6a94e7d37d32b806f5272a6e3e6b737518e669fe69cc4\": rpc error: code = NotFound desc = could not find container \"404e07f6337805ed03c6a94e7d37d32b806f5272a6e3e6b737518e669fe69cc4\": container with ID starting with 404e07f6337805ed03c6a94e7d37d32b806f5272a6e3e6b737518e669fe69cc4 not found: ID does not exist"
Jul 28 17:56:40.474579 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.474539    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"]
Jul 28 17:56:40.485352 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.485316    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4nps\" (UniqueName: \"kubernetes.io/projected/38d6c714-235e-47f5-8bfe-e4440d02cd35-kube-api-access-z4nps\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:56:40.485352 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.485349    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d6c714-235e-47f5-8bfe-e4440d02cd35-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:56:40.485579 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:40.485413    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkjhcn"]
Jul 28 17:56:42.312777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:42.312730    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38d6c714-235e-47f5-8bfe-e4440d02cd35" path="/var/lib/kubelet/pods/38d6c714-235e-47f5-8bfe-e4440d02cd35/volumes"
Jul 28 17:56:44.361168 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:44.361140    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:56:44.367662 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:56:44.367634    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 17:57:08.083901 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.083802    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"]
Jul 28 17:57:08.084559 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.084198    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d9adf4e-d6df-4481-84fd-515694f35fb4" containerName="llm-d-routing-sidecar"
Jul 28 17:57:08.084559 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.084210    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d9adf4e-d6df-4481-84fd-515694f35fb4" containerName="llm-d-routing-sidecar"
Jul 28 17:57:08.084559 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.084233    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d9adf4e-d6df-4481-84fd-515694f35fb4" containerName="main"
Jul 28 17:57:08.084559 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.084238    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d9adf4e-d6df-4481-84fd-515694f35fb4" containerName="main"
Jul 28 17:57:08.084559 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.084247    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38d6c714-235e-47f5-8bfe-e4440d02cd35" containerName="main"
Jul 28 17:57:08.084559 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.084253    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="38d6c714-235e-47f5-8bfe-e4440d02cd35" containerName="main"
Jul 28 17:57:08.084559 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.084309    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d9adf4e-d6df-4481-84fd-515694f35fb4" containerName="main"
Jul 28 17:57:08.084559 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.084316    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d9adf4e-d6df-4481-84fd-515694f35fb4" containerName="llm-d-routing-sidecar"
Jul 28 17:57:08.084559 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.084323    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="38d6c714-235e-47f5-8bfe-e4440d02cd35" containerName="main"
Jul 28 17:57:08.087462 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.087437    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"
Jul 28 17:57:08.091043 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.091010    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-8vd9t\""
Jul 28 17:57:08.091192 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.091045    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 28 17:57:08.091192 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.091019    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 28 17:57:08.091192 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.091129    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 28 17:57:08.091367 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.091227    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-42pqb\""
Jul 28 17:57:08.099882 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.099851    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"]
Jul 28 17:57:08.137980 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.137931    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5f616d-65b2-4724-ba0e-26f43d428cad-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc\" (UID: \"3b5f616d-65b2-4724-ba0e-26f43d428cad\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"
Jul 28 17:57:08.138176 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.137994    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7lrk\" (UniqueName: \"kubernetes.io/projected/3b5f616d-65b2-4724-ba0e-26f43d428cad-kube-api-access-s7lrk\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc\" (UID: \"3b5f616d-65b2-4724-ba0e-26f43d428cad\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"
Jul 28 17:57:08.238791 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.238746    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5f616d-65b2-4724-ba0e-26f43d428cad-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc\" (UID: \"3b5f616d-65b2-4724-ba0e-26f43d428cad\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"
Jul 28 17:57:08.239001 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.238810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7lrk\" (UniqueName: \"kubernetes.io/projected/3b5f616d-65b2-4724-ba0e-26f43d428cad-kube-api-access-s7lrk\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc\" (UID: \"3b5f616d-65b2-4724-ba0e-26f43d428cad\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"
Jul 28 17:57:08.241405 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.241379    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5f616d-65b2-4724-ba0e-26f43d428cad-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc\" (UID: \"3b5f616d-65b2-4724-ba0e-26f43d428cad\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"
Jul 28 17:57:08.248271 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.248239    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7lrk\" (UniqueName: \"kubernetes.io/projected/3b5f616d-65b2-4724-ba0e-26f43d428cad-kube-api-access-s7lrk\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc\" (UID: \"3b5f616d-65b2-4724-ba0e-26f43d428cad\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"
Jul 28 17:57:08.398096 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.397988    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"
Jul 28 17:57:08.554963 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.554935    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"]
Jul 28 17:57:08.557646 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:57:08.557598    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5f616d_65b2_4724_ba0e_26f43d428cad.slice/crio-5219f4ce7d813ed75b8aac14405bde4552d43430ae6a7fb5585e3945454fc3fc WatchSource:0}: Error finding container 5219f4ce7d813ed75b8aac14405bde4552d43430ae6a7fb5585e3945454fc3fc: Status 404 returned error can't find the container with id 5219f4ce7d813ed75b8aac14405bde4552d43430ae6a7fb5585e3945454fc3fc
Jul 28 17:57:08.592805 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:08.592775    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc" event={"ID":"3b5f616d-65b2-4724-ba0e-26f43d428cad","Type":"ContainerStarted","Data":"5219f4ce7d813ed75b8aac14405bde4552d43430ae6a7fb5585e3945454fc3fc"}
Jul 28 17:57:09.598599 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:09.598558    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc" event={"ID":"3b5f616d-65b2-4724-ba0e-26f43d428cad","Type":"ContainerStarted","Data":"4fe4ae4d4574bcfcd220bd48f7c73fc54bd1ed2b2e17c3da34ac2f74ef6c11a8"}
Jul 28 17:57:09.599014 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:09.598767    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"
Jul 28 17:57:09.617800 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:09.617743    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc" podStartSLOduration=1.617724549 podStartE2EDuration="1.617724549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:09.616618382 +0000 UTC m=+2425.874228071" watchObservedRunningTime="2026-07-28 17:57:09.617724549 +0000 UTC m=+2425.875334349"
Jul 28 17:57:40.605931 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:57:40.605899    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"
Jul 28 17:58:20.605893 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:20.605842    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc" podUID="3b5f616d-65b2-4724-ba0e-26f43d428cad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:58:20.658600 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:20.658564    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"]
Jul 28 17:58:30.605059 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:30.605018    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc" podUID="3b5f616d-65b2-4724-ba0e-26f43d428cad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:58:35.683516 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:35.683452    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc" podUID="3b5f616d-65b2-4724-ba0e-26f43d428cad" containerName="main" containerID="cri-o://4fe4ae4d4574bcfcd220bd48f7c73fc54bd1ed2b2e17c3da34ac2f74ef6c11a8" gracePeriod=45
Jul 28 17:58:35.944464 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:35.944405    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"
Jul 28 17:58:36.008409 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:36.008376    2575 generic.go:356] "Generic (PLEG): container finished" podID="3b5f616d-65b2-4724-ba0e-26f43d428cad" containerID="4fe4ae4d4574bcfcd220bd48f7c73fc54bd1ed2b2e17c3da34ac2f74ef6c11a8" exitCode=0
Jul 28 17:58:36.008625 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:36.008420    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"
Jul 28 17:58:36.008625 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:36.008451    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc" event={"ID":"3b5f616d-65b2-4724-ba0e-26f43d428cad","Type":"ContainerDied","Data":"4fe4ae4d4574bcfcd220bd48f7c73fc54bd1ed2b2e17c3da34ac2f74ef6c11a8"}
Jul 28 17:58:36.008625 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:36.008504    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc" event={"ID":"3b5f616d-65b2-4724-ba0e-26f43d428cad","Type":"ContainerDied","Data":"5219f4ce7d813ed75b8aac14405bde4552d43430ae6a7fb5585e3945454fc3fc"}
Jul 28 17:58:36.008625 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:36.008546    2575 scope.go:117] "RemoveContainer" containerID="4fe4ae4d4574bcfcd220bd48f7c73fc54bd1ed2b2e17c3da34ac2f74ef6c11a8"
Jul 28 17:58:36.016990 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:36.016972    2575 scope.go:117] "RemoveContainer" containerID="4fe4ae4d4574bcfcd220bd48f7c73fc54bd1ed2b2e17c3da34ac2f74ef6c11a8"
Jul 28 17:58:36.017253 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:58:36.017227    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fe4ae4d4574bcfcd220bd48f7c73fc54bd1ed2b2e17c3da34ac2f74ef6c11a8\": container with ID starting with 4fe4ae4d4574bcfcd220bd48f7c73fc54bd1ed2b2e17c3da34ac2f74ef6c11a8 not found: ID does not exist" containerID="4fe4ae4d4574bcfcd220bd48f7c73fc54bd1ed2b2e17c3da34ac2f74ef6c11a8"
Jul 28 17:58:36.017301 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:36.017264    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe4ae4d4574bcfcd220bd48f7c73fc54bd1ed2b2e17c3da34ac2f74ef6c11a8"} err="failed to get container status \"4fe4ae4d4574bcfcd220bd48f7c73fc54bd1ed2b2e17c3da34ac2f74ef6c11a8\": rpc error: code = NotFound desc = could not find container \"4fe4ae4d4574bcfcd220bd48f7c73fc54bd1ed2b2e17c3da34ac2f74ef6c11a8\": container with ID starting with 4fe4ae4d4574bcfcd220bd48f7c73fc54bd1ed2b2e17c3da34ac2f74ef6c11a8 not found: ID does not exist"
Jul 28 17:58:36.018557 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:36.018516    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5f616d-65b2-4724-ba0e-26f43d428cad-tls-certs\") pod \"3b5f616d-65b2-4724-ba0e-26f43d428cad\" (UID: \"3b5f616d-65b2-4724-ba0e-26f43d428cad\") "
Jul 28 17:58:36.018625 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:36.018569    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7lrk\" (UniqueName: \"kubernetes.io/projected/3b5f616d-65b2-4724-ba0e-26f43d428cad-kube-api-access-s7lrk\") pod \"3b5f616d-65b2-4724-ba0e-26f43d428cad\" (UID: \"3b5f616d-65b2-4724-ba0e-26f43d428cad\") "
Jul 28 17:58:36.020825 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:36.020802    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5f616d-65b2-4724-ba0e-26f43d428cad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b5f616d-65b2-4724-ba0e-26f43d428cad" (UID: "3b5f616d-65b2-4724-ba0e-26f43d428cad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:58:36.020988 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:36.020966    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5f616d-65b2-4724-ba0e-26f43d428cad-kube-api-access-s7lrk" (OuterVolumeSpecName: "kube-api-access-s7lrk") pod "3b5f616d-65b2-4724-ba0e-26f43d428cad" (UID: "3b5f616d-65b2-4724-ba0e-26f43d428cad"). InnerVolumeSpecName "kube-api-access-s7lrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:58:36.119730 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:36.119683    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5f616d-65b2-4724-ba0e-26f43d428cad-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:58:36.119730 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:36.119719    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7lrk\" (UniqueName: \"kubernetes.io/projected/3b5f616d-65b2-4724-ba0e-26f43d428cad-kube-api-access-s7lrk\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:58:36.334199 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:36.334165    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"]
Jul 28 17:58:36.339786 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:36.339760    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65nwmfc"]
Jul 28 17:58:38.312506 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:38.312466    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b5f616d-65b2-4724-ba0e-26f43d428cad" path="/var/lib/kubelet/pods/3b5f616d-65b2-4724-ba0e-26f43d428cad/volumes"
Jul 28 17:58:45.138405 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.138366    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"]
Jul 28 17:58:45.138799 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.138724    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b5f616d-65b2-4724-ba0e-26f43d428cad" containerName="main"
Jul 28 17:58:45.138799 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.138736    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b5f616d-65b2-4724-ba0e-26f43d428cad" containerName="main"
Jul 28 17:58:45.138877 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.138803    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b5f616d-65b2-4724-ba0e-26f43d428cad" containerName="main"
Jul 28 17:58:45.142994 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.142973    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"
Jul 28 17:58:45.145781 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.145760    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 28 17:58:45.145874 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.145804    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-fxwf7\""
Jul 28 17:58:45.146652 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.146631    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 28 17:58:45.146735 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.146688    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 28 17:58:45.147050 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.147035    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-wtbnx\""
Jul 28 17:58:45.158303 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.158280    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"]
Jul 28 17:58:45.194781 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.194738    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vjrn\" (UniqueName: \"kubernetes.io/projected/2d32614a-8314-4f42-a30b-362097169a0d-kube-api-access-7vjrn\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p\" (UID: \"2d32614a-8314-4f42-a30b-362097169a0d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"
Jul 28 17:58:45.194963 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.194867    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d32614a-8314-4f42-a30b-362097169a0d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p\" (UID: \"2d32614a-8314-4f42-a30b-362097169a0d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"
Jul 28 17:58:45.295354 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.295308    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d32614a-8314-4f42-a30b-362097169a0d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p\" (UID: \"2d32614a-8314-4f42-a30b-362097169a0d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"
Jul 28 17:58:45.295828 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.295392    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vjrn\" (UniqueName: \"kubernetes.io/projected/2d32614a-8314-4f42-a30b-362097169a0d-kube-api-access-7vjrn\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p\" (UID: \"2d32614a-8314-4f42-a30b-362097169a0d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"
Jul 28 17:58:45.297968 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.297943    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d32614a-8314-4f42-a30b-362097169a0d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p\" (UID: \"2d32614a-8314-4f42-a30b-362097169a0d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"
Jul 28 17:58:45.307297 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.307274    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vjrn\" (UniqueName: \"kubernetes.io/projected/2d32614a-8314-4f42-a30b-362097169a0d-kube-api-access-7vjrn\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p\" (UID: \"2d32614a-8314-4f42-a30b-362097169a0d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"
Jul 28 17:58:45.453334 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.453304    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"
Jul 28 17:58:45.592031 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:45.591888    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"]
Jul 28 17:58:45.594592 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:58:45.594563    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d32614a_8314_4f42_a30b_362097169a0d.slice/crio-0d9147ca5bf9dd891601ddf5be6862b8747ea13c8d67138f8c2f342c690519af WatchSource:0}: Error finding container 0d9147ca5bf9dd891601ddf5be6862b8747ea13c8d67138f8c2f342c690519af: Status 404 returned error can't find the container with id 0d9147ca5bf9dd891601ddf5be6862b8747ea13c8d67138f8c2f342c690519af
Jul 28 17:58:46.059489 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:46.059455    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p" event={"ID":"2d32614a-8314-4f42-a30b-362097169a0d","Type":"ContainerStarted","Data":"25d4e9a6868087991887a5f812f29e8e54d83a10c633a3358307e4f78f613231"}
Jul 28 17:58:46.059694 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:46.059513    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p" event={"ID":"2d32614a-8314-4f42-a30b-362097169a0d","Type":"ContainerStarted","Data":"0d9147ca5bf9dd891601ddf5be6862b8747ea13c8d67138f8c2f342c690519af"}
Jul 28 17:58:46.059694 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:46.059582    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"
Jul 28 17:58:46.084395 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:58:46.084346    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p" podStartSLOduration=1.084331515 podStartE2EDuration="1.084331515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:46.08215638 +0000 UTC m=+2522.339766067" watchObservedRunningTime="2026-07-28 17:58:46.084331515 +0000 UTC m=+2522.341941205"
Jul 28 17:59:17.066397 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:17.066362    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"
Jul 28 17:59:23.845704 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:23.845665    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"]
Jul 28 17:59:27.067027 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:27.066988    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p" podUID="2d32614a-8314-4f42-a30b-362097169a0d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:59:37.066473 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:37.066428    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p" podUID="2d32614a-8314-4f42-a30b-362097169a0d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:59:38.867889 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:38.867820    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p" podUID="2d32614a-8314-4f42-a30b-362097169a0d" containerName="main" containerID="cri-o://25d4e9a6868087991887a5f812f29e8e54d83a10c633a3358307e4f78f613231" gracePeriod=45
Jul 28 17:59:39.122003 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.121939    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"
Jul 28 17:59:39.271967 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.271929    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d32614a-8314-4f42-a30b-362097169a0d-tls-certs\") pod \"2d32614a-8314-4f42-a30b-362097169a0d\" (UID: \"2d32614a-8314-4f42-a30b-362097169a0d\") "
Jul 28 17:59:39.272123 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.272023    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vjrn\" (UniqueName: \"kubernetes.io/projected/2d32614a-8314-4f42-a30b-362097169a0d-kube-api-access-7vjrn\") pod \"2d32614a-8314-4f42-a30b-362097169a0d\" (UID: \"2d32614a-8314-4f42-a30b-362097169a0d\") "
Jul 28 17:59:39.274145 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.274113    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d32614a-8314-4f42-a30b-362097169a0d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d32614a-8314-4f42-a30b-362097169a0d" (UID: "2d32614a-8314-4f42-a30b-362097169a0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:59:39.274260 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.274190    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d32614a-8314-4f42-a30b-362097169a0d-kube-api-access-7vjrn" (OuterVolumeSpecName: "kube-api-access-7vjrn") pod "2d32614a-8314-4f42-a30b-362097169a0d" (UID: "2d32614a-8314-4f42-a30b-362097169a0d"). InnerVolumeSpecName "kube-api-access-7vjrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:59:39.300438 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.300406    2575 generic.go:356] "Generic (PLEG): container finished" podID="2d32614a-8314-4f42-a30b-362097169a0d" containerID="25d4e9a6868087991887a5f812f29e8e54d83a10c633a3358307e4f78f613231" exitCode=0
Jul 28 17:59:39.300564 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.300449    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"
Jul 28 17:59:39.300564 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.300492    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p" event={"ID":"2d32614a-8314-4f42-a30b-362097169a0d","Type":"ContainerDied","Data":"25d4e9a6868087991887a5f812f29e8e54d83a10c633a3358307e4f78f613231"}
Jul 28 17:59:39.300564 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.300557    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p" event={"ID":"2d32614a-8314-4f42-a30b-362097169a0d","Type":"ContainerDied","Data":"0d9147ca5bf9dd891601ddf5be6862b8747ea13c8d67138f8c2f342c690519af"}
Jul 28 17:59:39.300679 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.300586    2575 scope.go:117] "RemoveContainer" containerID="25d4e9a6868087991887a5f812f29e8e54d83a10c633a3358307e4f78f613231"
Jul 28 17:59:39.311053 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.311026    2575 scope.go:117] "RemoveContainer" containerID="25d4e9a6868087991887a5f812f29e8e54d83a10c633a3358307e4f78f613231"
Jul 28 17:59:39.311329 ip-10-0-135-188 kubenswrapper[2575]: E0728 17:59:39.311309    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d4e9a6868087991887a5f812f29e8e54d83a10c633a3358307e4f78f613231\": container with ID starting with 25d4e9a6868087991887a5f812f29e8e54d83a10c633a3358307e4f78f613231 not found: ID does not exist" containerID="25d4e9a6868087991887a5f812f29e8e54d83a10c633a3358307e4f78f613231"
Jul 28 17:59:39.311401 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.311339    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d4e9a6868087991887a5f812f29e8e54d83a10c633a3358307e4f78f613231"} err="failed to get container status \"25d4e9a6868087991887a5f812f29e8e54d83a10c633a3358307e4f78f613231\": rpc error: code = NotFound desc = could not find container \"25d4e9a6868087991887a5f812f29e8e54d83a10c633a3358307e4f78f613231\": container with ID starting with 25d4e9a6868087991887a5f812f29e8e54d83a10c633a3358307e4f78f613231 not found: ID does not exist"
Jul 28 17:59:39.323777 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.323747    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"]
Jul 28 17:59:39.327855 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.327828    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9lvb8p"]
Jul 28 17:59:39.373741 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.373663    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d32614a-8314-4f42-a30b-362097169a0d-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:59:39.373741 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:39.373686    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vjrn\" (UniqueName: \"kubernetes.io/projected/2d32614a-8314-4f42-a30b-362097169a0d-kube-api-access-7vjrn\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 17:59:40.312660 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:40.312628    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d32614a-8314-4f42-a30b-362097169a0d" path="/var/lib/kubelet/pods/2d32614a-8314-4f42-a30b-362097169a0d/volumes"
Jul 28 17:59:52.110592 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.110559    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"]
Jul 28 17:59:52.110988 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.110891    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d32614a-8314-4f42-a30b-362097169a0d" containerName="main"
Jul 28 17:59:52.110988 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.110903    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d32614a-8314-4f42-a30b-362097169a0d" containerName="main"
Jul 28 17:59:52.110988 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.110981    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d32614a-8314-4f42-a30b-362097169a0d" containerName="main"
Jul 28 17:59:52.113856 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.113840    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.117217 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.117192    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 28 17:59:52.117365 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.117199    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-f8g4c\""
Jul 28 17:59:52.117453 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.117370    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 28 17:59:52.117453 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.117430    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 28 17:59:52.125473 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.125451    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"]
Jul 28 17:59:52.283902 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.283858    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.283902 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.283898    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.284163 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.284008    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/916c3ab8-c880-4118-9bfe-49d507baab4d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.284163 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.284089    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.284163 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.284126    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vzr2\" (UniqueName: \"kubernetes.io/projected/916c3ab8-c880-4118-9bfe-49d507baab4d-kube-api-access-6vzr2\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.284163 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.284155    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.385072 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.384985    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.385072 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.385032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vzr2\" (UniqueName: \"kubernetes.io/projected/916c3ab8-c880-4118-9bfe-49d507baab4d-kube-api-access-6vzr2\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.385072 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.385061    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.385354 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.385107    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.385354 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.385130    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.385354 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.385206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/916c3ab8-c880-4118-9bfe-49d507baab4d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.385515 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.385449    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.385515 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.385487    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.385515 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.385509    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.387511 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.387489    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.387769 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.387750    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/916c3ab8-c880-4118-9bfe-49d507baab4d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.394315 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.394286    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vzr2\" (UniqueName: \"kubernetes.io/projected/916c3ab8-c880-4118-9bfe-49d507baab4d-kube-api-access-6vzr2\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.425027 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.424995    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 17:59:52.575794 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:52.575759    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"]
Jul 28 17:59:52.577892 ip-10-0-135-188 kubenswrapper[2575]: W0728 17:59:52.577849    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916c3ab8_c880_4118_9bfe_49d507baab4d.slice/crio-446cb39c38cbc0700a2e276016417a5fd1399d5293f2d2488d3a1beada4393da WatchSource:0}: Error finding container 446cb39c38cbc0700a2e276016417a5fd1399d5293f2d2488d3a1beada4393da: Status 404 returned error can't find the container with id 446cb39c38cbc0700a2e276016417a5fd1399d5293f2d2488d3a1beada4393da
Jul 28 17:59:53.371372 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:53.371340    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8" event={"ID":"916c3ab8-c880-4118-9bfe-49d507baab4d","Type":"ContainerStarted","Data":"b75704c5f1bd177c26839316baa16d2d3fe00008de92f79969240bc1805f2411"}
Jul 28 17:59:53.371765 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:53.371392    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8" event={"ID":"916c3ab8-c880-4118-9bfe-49d507baab4d","Type":"ContainerStarted","Data":"446cb39c38cbc0700a2e276016417a5fd1399d5293f2d2488d3a1beada4393da"}
Jul 28 17:59:53.389575 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:53.389504    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8" podStartSLOduration=1.389490148 podStartE2EDuration="1.389490148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:59:53.388781087 +0000 UTC m=+2589.646390777" watchObservedRunningTime="2026-07-28 17:59:53.389490148 +0000 UTC m=+2589.647099838"
Jul 28 17:59:53.426022 ip-10-0-135-188 kubenswrapper[2575]: I0728 17:59:53.425997    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 18:00:02.425569 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:02.425532    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 18:00:02.440438 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:02.440409    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 18:00:03.427682 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:03.427654    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 18:00:25.916057 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:25.915974    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"]
Jul 28 18:00:25.925815 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:00:25.925780    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:00:25.925815 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:00:25.925815 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:00:25.925815 ip-10-0-135-188 kubenswrapper[2575]:  > containerID="b75704c5f1bd177c26839316baa16d2d3fe00008de92f79969240bc1805f2411" cmd=["/bin/sleep","15"]
Jul 28 18:00:25.926072 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:25.925831    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8" message="" err=<
Jul 28 18:00:25.926072 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:00:25.926072 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:00:25.926072 ip-10-0-135-188 kubenswrapper[2575]:  >
Jul 28 18:00:25.926072 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:00:25.925843    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:00:25.926072 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:00:25.926072 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:00:25.926072 ip-10-0-135-188 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8" podUID="916c3ab8-c880-4118-9bfe-49d507baab4d" containerName="main" containerID="cri-o://b75704c5f1bd177c26839316baa16d2d3fe00008de92f79969240bc1805f2411"
Jul 28 18:00:25.926072 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:25.925889    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8" podUID="916c3ab8-c880-4118-9bfe-49d507baab4d" containerName="main" containerID="cri-o://b75704c5f1bd177c26839316baa16d2d3fe00008de92f79969240bc1805f2411" gracePeriod=60
Jul 28 18:00:26.282213 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.282185    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 18:00:26.377020 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.376984    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-tmp-dir\") pod \"916c3ab8-c880-4118-9bfe-49d507baab4d\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") "
Jul 28 18:00:26.377193 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.377074    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/916c3ab8-c880-4118-9bfe-49d507baab4d-tls-certs\") pod \"916c3ab8-c880-4118-9bfe-49d507baab4d\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") "
Jul 28 18:00:26.377193 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.377125    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-dshm\") pod \"916c3ab8-c880-4118-9bfe-49d507baab4d\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") "
Jul 28 18:00:26.377193 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.377148    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vzr2\" (UniqueName: \"kubernetes.io/projected/916c3ab8-c880-4118-9bfe-49d507baab4d-kube-api-access-6vzr2\") pod \"916c3ab8-c880-4118-9bfe-49d507baab4d\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") "
Jul 28 18:00:26.377354 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.377271    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-home\") pod \"916c3ab8-c880-4118-9bfe-49d507baab4d\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") "
Jul 28 18:00:26.377354 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.377318    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-model-cache\") pod \"916c3ab8-c880-4118-9bfe-49d507baab4d\" (UID: \"916c3ab8-c880-4118-9bfe-49d507baab4d\") "
Jul 28 18:00:26.377354 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.377279    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "916c3ab8-c880-4118-9bfe-49d507baab4d" (UID: "916c3ab8-c880-4118-9bfe-49d507baab4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:26.377496 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.377453    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-home" (OuterVolumeSpecName: "home") pod "916c3ab8-c880-4118-9bfe-49d507baab4d" (UID: "916c3ab8-c880-4118-9bfe-49d507baab4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:26.377645 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.377616    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-model-cache" (OuterVolumeSpecName: "model-cache") pod "916c3ab8-c880-4118-9bfe-49d507baab4d" (UID: "916c3ab8-c880-4118-9bfe-49d507baab4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:26.377730 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.377664    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-tmp-dir\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:00:26.377730 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.377682    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-home\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:00:26.379445 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.379423    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916c3ab8-c880-4118-9bfe-49d507baab4d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "916c3ab8-c880-4118-9bfe-49d507baab4d" (UID: "916c3ab8-c880-4118-9bfe-49d507baab4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:00:26.379751 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.379723    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-dshm" (OuterVolumeSpecName: "dshm") pod "916c3ab8-c880-4118-9bfe-49d507baab4d" (UID: "916c3ab8-c880-4118-9bfe-49d507baab4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:26.379830 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.379732    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916c3ab8-c880-4118-9bfe-49d507baab4d-kube-api-access-6vzr2" (OuterVolumeSpecName: "kube-api-access-6vzr2") pod "916c3ab8-c880-4118-9bfe-49d507baab4d" (UID: "916c3ab8-c880-4118-9bfe-49d507baab4d"). InnerVolumeSpecName "kube-api-access-6vzr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:00:26.478295 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.478192    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/916c3ab8-c880-4118-9bfe-49d507baab4d-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:00:26.478295 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.478225    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-dshm\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:00:26.478295 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.478239    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vzr2\" (UniqueName: \"kubernetes.io/projected/916c3ab8-c880-4118-9bfe-49d507baab4d-kube-api-access-6vzr2\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:00:26.478295 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.478255    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/916c3ab8-c880-4118-9bfe-49d507baab4d-model-cache\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:00:26.527367 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.527336    2575 generic.go:356] "Generic (PLEG): container finished" podID="916c3ab8-c880-4118-9bfe-49d507baab4d" containerID="b75704c5f1bd177c26839316baa16d2d3fe00008de92f79969240bc1805f2411" exitCode=0
Jul 28 18:00:26.527555 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.527379    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"
Jul 28 18:00:26.527555 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.527418    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8" event={"ID":"916c3ab8-c880-4118-9bfe-49d507baab4d","Type":"ContainerDied","Data":"b75704c5f1bd177c26839316baa16d2d3fe00008de92f79969240bc1805f2411"}
Jul 28 18:00:26.527555 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.527462    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8" event={"ID":"916c3ab8-c880-4118-9bfe-49d507baab4d","Type":"ContainerDied","Data":"446cb39c38cbc0700a2e276016417a5fd1399d5293f2d2488d3a1beada4393da"}
Jul 28 18:00:26.527555 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.527488    2575 scope.go:117] "RemoveContainer" containerID="b75704c5f1bd177c26839316baa16d2d3fe00008de92f79969240bc1805f2411"
Jul 28 18:00:26.537192 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.537169    2575 scope.go:117] "RemoveContainer" containerID="b75704c5f1bd177c26839316baa16d2d3fe00008de92f79969240bc1805f2411"
Jul 28 18:00:26.537469 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:00:26.537450    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75704c5f1bd177c26839316baa16d2d3fe00008de92f79969240bc1805f2411\": container with ID starting with b75704c5f1bd177c26839316baa16d2d3fe00008de92f79969240bc1805f2411 not found: ID does not exist" containerID="b75704c5f1bd177c26839316baa16d2d3fe00008de92f79969240bc1805f2411"
Jul 28 18:00:26.537558 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.537482    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75704c5f1bd177c26839316baa16d2d3fe00008de92f79969240bc1805f2411"} err="failed to get container status \"b75704c5f1bd177c26839316baa16d2d3fe00008de92f79969240bc1805f2411\": rpc error: code = NotFound desc = could not find container \"b75704c5f1bd177c26839316baa16d2d3fe00008de92f79969240bc1805f2411\": container with ID starting with b75704c5f1bd177c26839316baa16d2d3fe00008de92f79969240bc1805f2411 not found: ID does not exist"
Jul 28 18:00:26.549573 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.549510    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"]
Jul 28 18:00:26.553861 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:26.553839    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6kt8"]
Jul 28 18:00:28.312689 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:28.312653    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="916c3ab8-c880-4118-9bfe-49d507baab4d" path="/var/lib/kubelet/pods/916c3ab8-c880-4118-9bfe-49d507baab4d/volumes"
Jul 28 18:00:51.009153 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.009117    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"]
Jul 28 18:00:51.009578 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.009463    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916c3ab8-c880-4118-9bfe-49d507baab4d" containerName="main"
Jul 28 18:00:51.009578 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.009474    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="916c3ab8-c880-4118-9bfe-49d507baab4d" containerName="main"
Jul 28 18:00:51.009578 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.009563    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="916c3ab8-c880-4118-9bfe-49d507baab4d" containerName="main"
Jul 28 18:00:51.014231 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.014208    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.017376 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.017351    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 28 18:00:51.017556 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.017355    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 28 18:00:51.017556 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.017353    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-n26hm\""
Jul 28 18:00:51.017556 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.017402    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 28 18:00:51.026488 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.026464    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"]
Jul 28 18:00:51.103126 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.103083    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.103126 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.103124    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.103348 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.103150    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.103348 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.103237    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.103348 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.103326    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.103446 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.103353    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f7bn\" (UniqueName: \"kubernetes.io/projected/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-kube-api-access-5f7bn\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.203852 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.203813    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.204102 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.203887    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.204102 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.203912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f7bn\" (UniqueName: \"kubernetes.io/projected/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-kube-api-access-5f7bn\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.204102 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.203945    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.204102 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.203962    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.204102 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.203985    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.204377 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.204264    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.204419 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.204369    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.204419 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.204400    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.206328 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.206304    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.206504 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.206483    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.220440 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.220399    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f7bn\" (UniqueName: \"kubernetes.io/projected/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-kube-api-access-5f7bn\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.325436 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.325346    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:00:51.495577 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:00:51.495543    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba000eb5_13fc_4e9a_ba92_559f9c405bc3.slice/crio-da400757e66111dfbcc72e77fa8321b4c49e7cb448d5a518ea5fb8488b322058 WatchSource:0}: Error finding container da400757e66111dfbcc72e77fa8321b4c49e7cb448d5a518ea5fb8488b322058: Status 404 returned error can't find the container with id da400757e66111dfbcc72e77fa8321b4c49e7cb448d5a518ea5fb8488b322058
Jul 28 18:00:51.495837 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.495816    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"]
Jul 28 18:00:51.639734 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.639644    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq" event={"ID":"ba000eb5-13fc-4e9a-ba92-559f9c405bc3","Type":"ContainerStarted","Data":"e842b700573d9557076160a0fb393a8f70b3a2e1fa271810f97cc2ae4399ca48"}
Jul 28 18:00:51.639734 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.639698    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq" event={"ID":"ba000eb5-13fc-4e9a-ba92-559f9c405bc3","Type":"ContainerStarted","Data":"da400757e66111dfbcc72e77fa8321b4c49e7cb448d5a518ea5fb8488b322058"}
Jul 28 18:00:51.690949 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:51.690898    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq" podStartSLOduration=1.690881982 podStartE2EDuration="1.690881982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:00:51.689232911 +0000 UTC m=+2647.946842615" watchObservedRunningTime="2026-07-28 18:00:51.690881982 +0000 UTC m=+2647.948491672"
Jul 28 18:00:52.325585 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:00:52.325542    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:01:01.325713 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:01.325668    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:01:01.338576 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:01.338540    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:01:01.704285 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:01.704254    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:01:44.388700 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:44.388656    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:01:44.397404 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:44.397378    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:01:50.706700 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.706665    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"]
Jul 28 18:01:50.710249 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.710226    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.712816 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.712796    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 28 18:01:50.722824 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.722803    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"]
Jul 28 18:01:50.836390 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.836347    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.836638 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.836418    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.836638 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.836471    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833f5558-c2d9-4a73-801a-7b2afabf5888-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.836638 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.836563    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.836638 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.836600    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc8mc\" (UniqueName: \"kubernetes.io/projected/833f5558-c2d9-4a73-801a-7b2afabf5888-kube-api-access-vc8mc\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.836819 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.836645    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.937847 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.937810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.938024 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.937869    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.938024 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.937921    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.938024 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.937956    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833f5558-c2d9-4a73-801a-7b2afabf5888-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.938193 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.938032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.938193 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.938075    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc8mc\" (UniqueName: \"kubernetes.io/projected/833f5558-c2d9-4a73-801a-7b2afabf5888-kube-api-access-vc8mc\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.938394 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.938364    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.938511 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.938431    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.938600 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.938560    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.940316 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.940291    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.940901 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.940885    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833f5558-c2d9-4a73-801a-7b2afabf5888-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:50.952652 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:50.952627    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc8mc\" (UniqueName: \"kubernetes.io/projected/833f5558-c2d9-4a73-801a-7b2afabf5888-kube-api-access-vc8mc\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:51.020355 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:51.020261    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:01:51.160700 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:51.160671    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"]
Jul 28 18:01:51.163277 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:01:51.163242    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833f5558_c2d9_4a73_801a_7b2afabf5888.slice/crio-2dfba439ec48a93420ed07277ed92b201e84bf97ab2d6cad4f1e9d1b2d7b8503 WatchSource:0}: Error finding container 2dfba439ec48a93420ed07277ed92b201e84bf97ab2d6cad4f1e9d1b2d7b8503: Status 404 returned error can't find the container with id 2dfba439ec48a93420ed07277ed92b201e84bf97ab2d6cad4f1e9d1b2d7b8503
Jul 28 18:01:51.931679 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:51.931636    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh" event={"ID":"833f5558-c2d9-4a73-801a-7b2afabf5888","Type":"ContainerStarted","Data":"318330c42105b8ad63cba10af25e757477a7200a48e9bec4e4f3faa5f8f81562"}
Jul 28 18:01:51.932081 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:51.931693    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh" event={"ID":"833f5558-c2d9-4a73-801a-7b2afabf5888","Type":"ContainerStarted","Data":"2dfba439ec48a93420ed07277ed92b201e84bf97ab2d6cad4f1e9d1b2d7b8503"}
Jul 28 18:01:51.952815 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:51.952765    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh" podStartSLOduration=1.9527492149999999 podStartE2EDuration="1.952749215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:51.950853858 +0000 UTC m=+2708.208463549" watchObservedRunningTime="2026-07-28 18:01:51.952749215 +0000 UTC m=+2708.210358904"
Jul 28 18:01:52.021248 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:01:52.021213    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:02:01.021141 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:01.021090    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:02:01.034891 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:01.034861    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:02:01.996239 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:01.996210    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:02:24.153254 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.153222    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"]
Jul 28 18:02:24.162577 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:02:24.162502    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:02:24.162577 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:02:24.162577 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:02:24.162577 ip-10-0-135-188 kubenswrapper[2575]:  > containerID="318330c42105b8ad63cba10af25e757477a7200a48e9bec4e4f3faa5f8f81562" cmd=["/bin/sleep","15"]
Jul 28 18:02:24.162850 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.162616    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh" message="" err=<
Jul 28 18:02:24.162850 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:02:24.162850 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:02:24.162850 ip-10-0-135-188 kubenswrapper[2575]:  >
Jul 28 18:02:24.162850 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:02:24.162633    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:02:24.162850 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:02:24.162850 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:02:24.162850 ip-10-0-135-188 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh" podUID="833f5558-c2d9-4a73-801a-7b2afabf5888" containerName="main" containerID="cri-o://318330c42105b8ad63cba10af25e757477a7200a48e9bec4e4f3faa5f8f81562"
Jul 28 18:02:24.162850 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.162695    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh" podUID="833f5558-c2d9-4a73-801a-7b2afabf5888" containerName="main" containerID="cri-o://318330c42105b8ad63cba10af25e757477a7200a48e9bec4e4f3faa5f8f81562" gracePeriod=60
Jul 28 18:02:24.401895 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.401872    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:02:24.438660 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.438634    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-tmp-dir\") pod \"833f5558-c2d9-4a73-801a-7b2afabf5888\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") "
Jul 28 18:02:24.438844 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.438679    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833f5558-c2d9-4a73-801a-7b2afabf5888-tls-certs\") pod \"833f5558-c2d9-4a73-801a-7b2afabf5888\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") "
Jul 28 18:02:24.438844 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.438699    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-home\") pod \"833f5558-c2d9-4a73-801a-7b2afabf5888\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") "
Jul 28 18:02:24.438844 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.438727    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-dshm\") pod \"833f5558-c2d9-4a73-801a-7b2afabf5888\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") "
Jul 28 18:02:24.438844 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.438767    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-model-cache\") pod \"833f5558-c2d9-4a73-801a-7b2afabf5888\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") "
Jul 28 18:02:24.438844 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.438809    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc8mc\" (UniqueName: \"kubernetes.io/projected/833f5558-c2d9-4a73-801a-7b2afabf5888-kube-api-access-vc8mc\") pod \"833f5558-c2d9-4a73-801a-7b2afabf5888\" (UID: \"833f5558-c2d9-4a73-801a-7b2afabf5888\") "
Jul 28 18:02:24.439137 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.439027    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-home" (OuterVolumeSpecName: "home") pod "833f5558-c2d9-4a73-801a-7b2afabf5888" (UID: "833f5558-c2d9-4a73-801a-7b2afabf5888"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:02:24.439198 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.439154    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-model-cache" (OuterVolumeSpecName: "model-cache") pod "833f5558-c2d9-4a73-801a-7b2afabf5888" (UID: "833f5558-c2d9-4a73-801a-7b2afabf5888"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:02:24.439254 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.439192    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-home\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:02:24.439254 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.439214    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "833f5558-c2d9-4a73-801a-7b2afabf5888" (UID: "833f5558-c2d9-4a73-801a-7b2afabf5888"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:02:24.441193 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.441158    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833f5558-c2d9-4a73-801a-7b2afabf5888-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "833f5558-c2d9-4a73-801a-7b2afabf5888" (UID: "833f5558-c2d9-4a73-801a-7b2afabf5888"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:02:24.441300 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.441223    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-dshm" (OuterVolumeSpecName: "dshm") pod "833f5558-c2d9-4a73-801a-7b2afabf5888" (UID: "833f5558-c2d9-4a73-801a-7b2afabf5888"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:02:24.441300 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.441238    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833f5558-c2d9-4a73-801a-7b2afabf5888-kube-api-access-vc8mc" (OuterVolumeSpecName: "kube-api-access-vc8mc") pod "833f5558-c2d9-4a73-801a-7b2afabf5888" (UID: "833f5558-c2d9-4a73-801a-7b2afabf5888"). InnerVolumeSpecName "kube-api-access-vc8mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:02:24.540591 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.540560    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-model-cache\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:02:24.540591 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.540588    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc8mc\" (UniqueName: \"kubernetes.io/projected/833f5558-c2d9-4a73-801a-7b2afabf5888-kube-api-access-vc8mc\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:02:24.540591 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.540597    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-tmp-dir\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:02:24.540812 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.540607    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833f5558-c2d9-4a73-801a-7b2afabf5888-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:02:24.540812 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:24.540615    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/833f5558-c2d9-4a73-801a-7b2afabf5888-dshm\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:02:25.094606 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:25.094569    2575 generic.go:356] "Generic (PLEG): container finished" podID="833f5558-c2d9-4a73-801a-7b2afabf5888" containerID="318330c42105b8ad63cba10af25e757477a7200a48e9bec4e4f3faa5f8f81562" exitCode=0
Jul 28 18:02:25.094757 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:25.094621    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"
Jul 28 18:02:25.094757 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:25.094639    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh" event={"ID":"833f5558-c2d9-4a73-801a-7b2afabf5888","Type":"ContainerDied","Data":"318330c42105b8ad63cba10af25e757477a7200a48e9bec4e4f3faa5f8f81562"}
Jul 28 18:02:25.094757 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:25.094685    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh" event={"ID":"833f5558-c2d9-4a73-801a-7b2afabf5888","Type":"ContainerDied","Data":"2dfba439ec48a93420ed07277ed92b201e84bf97ab2d6cad4f1e9d1b2d7b8503"}
Jul 28 18:02:25.094757 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:25.094706    2575 scope.go:117] "RemoveContainer" containerID="318330c42105b8ad63cba10af25e757477a7200a48e9bec4e4f3faa5f8f81562"
Jul 28 18:02:25.103832 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:25.103813    2575 scope.go:117] "RemoveContainer" containerID="318330c42105b8ad63cba10af25e757477a7200a48e9bec4e4f3faa5f8f81562"
Jul 28 18:02:25.104065 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:02:25.104047    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318330c42105b8ad63cba10af25e757477a7200a48e9bec4e4f3faa5f8f81562\": container with ID starting with 318330c42105b8ad63cba10af25e757477a7200a48e9bec4e4f3faa5f8f81562 not found: ID does not exist" containerID="318330c42105b8ad63cba10af25e757477a7200a48e9bec4e4f3faa5f8f81562"
Jul 28 18:02:25.104114 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:25.104073    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318330c42105b8ad63cba10af25e757477a7200a48e9bec4e4f3faa5f8f81562"} err="failed to get container status \"318330c42105b8ad63cba10af25e757477a7200a48e9bec4e4f3faa5f8f81562\": rpc error: code = NotFound desc = could not find container \"318330c42105b8ad63cba10af25e757477a7200a48e9bec4e4f3faa5f8f81562\": container with ID starting with 318330c42105b8ad63cba10af25e757477a7200a48e9bec4e4f3faa5f8f81562 not found: ID does not exist"
Jul 28 18:02:25.118753 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:25.118726    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"]
Jul 28 18:02:25.124276 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:25.124255    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959g9mh"]
Jul 28 18:02:26.312308 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:26.312267    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="833f5558-c2d9-4a73-801a-7b2afabf5888" path="/var/lib/kubelet/pods/833f5558-c2d9-4a73-801a-7b2afabf5888/volumes"
Jul 28 18:02:44.082266 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.082229    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"]
Jul 28 18:02:44.091218 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:02:44.091182    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:02:44.091218 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:02:44.091218 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:02:44.091218 ip-10-0-135-188 kubenswrapper[2575]:  > containerID="e842b700573d9557076160a0fb393a8f70b3a2e1fa271810f97cc2ae4399ca48" cmd=["/bin/sleep","15"]
Jul 28 18:02:44.091538 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.091233    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq" message="" err=<
Jul 28 18:02:44.091538 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:02:44.091538 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:02:44.091538 ip-10-0-135-188 kubenswrapper[2575]:  >
Jul 28 18:02:44.091538 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:02:44.091246    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:02:44.091538 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:02:44.091538 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:02:44.091538 ip-10-0-135-188 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq" podUID="ba000eb5-13fc-4e9a-ba92-559f9c405bc3" containerName="main" containerID="cri-o://e842b700573d9557076160a0fb393a8f70b3a2e1fa271810f97cc2ae4399ca48"
Jul 28 18:02:44.091538 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.091289    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq" podUID="ba000eb5-13fc-4e9a-ba92-559f9c405bc3" containerName="main" containerID="cri-o://e842b700573d9557076160a0fb393a8f70b3a2e1fa271810f97cc2ae4399ca48" gracePeriod=60
Jul 28 18:02:44.448813 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.448790    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:02:44.514556 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.514505    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-dshm\") pod \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") "
Jul 28 18:02:44.514743 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.514576    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f7bn\" (UniqueName: \"kubernetes.io/projected/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-kube-api-access-5f7bn\") pod \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") "
Jul 28 18:02:44.514743 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.514624    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-tmp-dir\") pod \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") "
Jul 28 18:02:44.514743 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.514659    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-home\") pod \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") "
Jul 28 18:02:44.514743 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.514688    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-tls-certs\") pod \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") "
Jul 28 18:02:44.514743 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.514719    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-model-cache\") pod \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\" (UID: \"ba000eb5-13fc-4e9a-ba92-559f9c405bc3\") "
Jul 28 18:02:44.515019 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.514958    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba000eb5-13fc-4e9a-ba92-559f9c405bc3" (UID: "ba000eb5-13fc-4e9a-ba92-559f9c405bc3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:02:44.515019 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.514975    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-home" (OuterVolumeSpecName: "home") pod "ba000eb5-13fc-4e9a-ba92-559f9c405bc3" (UID: "ba000eb5-13fc-4e9a-ba92-559f9c405bc3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:02:44.515115 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.515059    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba000eb5-13fc-4e9a-ba92-559f9c405bc3" (UID: "ba000eb5-13fc-4e9a-ba92-559f9c405bc3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:02:44.516783 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.516743    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba000eb5-13fc-4e9a-ba92-559f9c405bc3" (UID: "ba000eb5-13fc-4e9a-ba92-559f9c405bc3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:02:44.516783 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.516777    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-kube-api-access-5f7bn" (OuterVolumeSpecName: "kube-api-access-5f7bn") pod "ba000eb5-13fc-4e9a-ba92-559f9c405bc3" (UID: "ba000eb5-13fc-4e9a-ba92-559f9c405bc3"). InnerVolumeSpecName "kube-api-access-5f7bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:02:44.516966 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.516785    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-dshm" (OuterVolumeSpecName: "dshm") pod "ba000eb5-13fc-4e9a-ba92-559f9c405bc3" (UID: "ba000eb5-13fc-4e9a-ba92-559f9c405bc3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:02:44.616018 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.615957    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-tmp-dir\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:02:44.616018 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.615983    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-home\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:02:44.616018 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.615992    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:02:44.616018 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.616001    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-model-cache\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:02:44.616018 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.616008    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-dshm\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:02:44.616018 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:44.616016    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f7bn\" (UniqueName: \"kubernetes.io/projected/ba000eb5-13fc-4e9a-ba92-559f9c405bc3-kube-api-access-5f7bn\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:02:45.189663 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:45.189615    2575 generic.go:356] "Generic (PLEG): container finished" podID="ba000eb5-13fc-4e9a-ba92-559f9c405bc3" containerID="e842b700573d9557076160a0fb393a8f70b3a2e1fa271810f97cc2ae4399ca48" exitCode=0
Jul 28 18:02:45.189663 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:45.189663    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"
Jul 28 18:02:45.190143 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:45.189690    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq" event={"ID":"ba000eb5-13fc-4e9a-ba92-559f9c405bc3","Type":"ContainerDied","Data":"e842b700573d9557076160a0fb393a8f70b3a2e1fa271810f97cc2ae4399ca48"}
Jul 28 18:02:45.190143 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:45.189733    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq" event={"ID":"ba000eb5-13fc-4e9a-ba92-559f9c405bc3","Type":"ContainerDied","Data":"da400757e66111dfbcc72e77fa8321b4c49e7cb448d5a518ea5fb8488b322058"}
Jul 28 18:02:45.190143 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:45.189755    2575 scope.go:117] "RemoveContainer" containerID="e842b700573d9557076160a0fb393a8f70b3a2e1fa271810f97cc2ae4399ca48"
Jul 28 18:02:45.199576 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:45.199556    2575 scope.go:117] "RemoveContainer" containerID="e842b700573d9557076160a0fb393a8f70b3a2e1fa271810f97cc2ae4399ca48"
Jul 28 18:02:45.199865 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:02:45.199841    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e842b700573d9557076160a0fb393a8f70b3a2e1fa271810f97cc2ae4399ca48\": container with ID starting with e842b700573d9557076160a0fb393a8f70b3a2e1fa271810f97cc2ae4399ca48 not found: ID does not exist" containerID="e842b700573d9557076160a0fb393a8f70b3a2e1fa271810f97cc2ae4399ca48"
Jul 28 18:02:45.199935 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:45.199874    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e842b700573d9557076160a0fb393a8f70b3a2e1fa271810f97cc2ae4399ca48"} err="failed to get container status \"e842b700573d9557076160a0fb393a8f70b3a2e1fa271810f97cc2ae4399ca48\": rpc error: code = NotFound desc = could not find container \"e842b700573d9557076160a0fb393a8f70b3a2e1fa271810f97cc2ae4399ca48\": container with ID starting with e842b700573d9557076160a0fb393a8f70b3a2e1fa271810f97cc2ae4399ca48 not found: ID does not exist"
Jul 28 18:02:45.212612 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:45.212589    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"]
Jul 28 18:02:45.216343 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:45.216321    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-ckvzq"]
Jul 28 18:02:46.312218 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:02:46.312175    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba000eb5-13fc-4e9a-ba92-559f9c405bc3" path="/var/lib/kubelet/pods/ba000eb5-13fc-4e9a-ba92-559f9c405bc3/volumes"
Jul 28 18:03:12.198179 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.198091    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"]
Jul 28 18:03:12.198738 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.198424    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833f5558-c2d9-4a73-801a-7b2afabf5888" containerName="main"
Jul 28 18:03:12.198738 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.198436    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="833f5558-c2d9-4a73-801a-7b2afabf5888" containerName="main"
Jul 28 18:03:12.198738 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.198454    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba000eb5-13fc-4e9a-ba92-559f9c405bc3" containerName="main"
Jul 28 18:03:12.198738 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.198459    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba000eb5-13fc-4e9a-ba92-559f9c405bc3" containerName="main"
Jul 28 18:03:12.198738 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.198535    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba000eb5-13fc-4e9a-ba92-559f9c405bc3" containerName="main"
Jul 28 18:03:12.198738 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.198547    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="833f5558-c2d9-4a73-801a-7b2afabf5888" containerName="main"
Jul 28 18:03:12.201498 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.201479    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.204566 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.204515    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 28 18:03:12.204693 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.204605    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 28 18:03:12.204693 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.204618    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-sbsdx\""
Jul 28 18:03:12.204693 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.204672    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 28 18:03:12.214582 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.214561    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"]
Jul 28 18:03:12.252374 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.252342    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z548m\" (UniqueName: \"kubernetes.io/projected/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-kube-api-access-z548m\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.252550 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.252400    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.252550 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.252432    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.252550 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.252465    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.252550 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.252543    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.252711 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.252633    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.353067 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.353030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z548m\" (UniqueName: \"kubernetes.io/projected/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-kube-api-access-z548m\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.353067 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.353071    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.353343 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.353099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.353343 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.353212    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.353343 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.353271    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.353514 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.353374    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.353514 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.353450    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.353709 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.353688    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.353771 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.353693    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.355559 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.355515    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.355734 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.355716    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.361240 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.361218    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z548m\" (UniqueName: \"kubernetes.io/projected/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-kube-api-access-z548m\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.511196 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.511098    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:12.573348 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.573309    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"]
Jul 28 18:03:12.577654 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.577625    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"
Jul 28 18:03:12.584092 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.584067    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-5l7l9\""
Jul 28 18:03:12.603095 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.603067    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"]
Jul 28 18:03:12.650408 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.650380    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"]
Jul 28 18:03:12.652883 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:03:12.652847    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594bced1_c2e9_45a4_8ec7_edc1c5e0399b.slice/crio-251daa7aa0469f9582d455fbe4b53b8b3497662ebed01d98378c9778948f1563 WatchSource:0}: Error finding container 251daa7aa0469f9582d455fbe4b53b8b3497662ebed01d98378c9778948f1563: Status 404 returned error can't find the container with id 251daa7aa0469f9582d455fbe4b53b8b3497662ebed01d98378c9778948f1563
Jul 28 18:03:12.654745 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.654722    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc2997be-3ec7-4520-b3ec-51477e2e3045-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7\" (UID: \"bc2997be-3ec7-4520-b3ec-51477e2e3045\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"
Jul 28 18:03:12.654865 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.654766    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsrpn\" (UniqueName: \"kubernetes.io/projected/bc2997be-3ec7-4520-b3ec-51477e2e3045-kube-api-access-rsrpn\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7\" (UID: \"bc2997be-3ec7-4520-b3ec-51477e2e3045\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"
Jul 28 18:03:12.756088 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.756055    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc2997be-3ec7-4520-b3ec-51477e2e3045-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7\" (UID: \"bc2997be-3ec7-4520-b3ec-51477e2e3045\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"
Jul 28 18:03:12.756236 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.756108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsrpn\" (UniqueName: \"kubernetes.io/projected/bc2997be-3ec7-4520-b3ec-51477e2e3045-kube-api-access-rsrpn\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7\" (UID: \"bc2997be-3ec7-4520-b3ec-51477e2e3045\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"
Jul 28 18:03:12.758562 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.758511    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc2997be-3ec7-4520-b3ec-51477e2e3045-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7\" (UID: \"bc2997be-3ec7-4520-b3ec-51477e2e3045\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"
Jul 28 18:03:12.764629 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.764570    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsrpn\" (UniqueName: \"kubernetes.io/projected/bc2997be-3ec7-4520-b3ec-51477e2e3045-kube-api-access-rsrpn\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7\" (UID: \"bc2997be-3ec7-4520-b3ec-51477e2e3045\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"
Jul 28 18:03:12.889150 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:12.889113    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"
Jul 28 18:03:13.020246 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:13.020217    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"]
Jul 28 18:03:13.022200 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:03:13.022172    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2997be_3ec7_4520_b3ec_51477e2e3045.slice/crio-fce23c729946522ce95d7377a01ea569dee2e498c59350f33365b0830e145adc WatchSource:0}: Error finding container fce23c729946522ce95d7377a01ea569dee2e498c59350f33365b0830e145adc: Status 404 returned error can't find the container with id fce23c729946522ce95d7377a01ea569dee2e498c59350f33365b0830e145adc
Jul 28 18:03:13.320654 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:13.320555    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8" event={"ID":"594bced1-c2e9-45a4-8ec7-edc1c5e0399b","Type":"ContainerStarted","Data":"a4f354ac13c1dc303c1ee779ac0bad802b4f0efaf8054c421277b92b49ebf9d3"}
Jul 28 18:03:13.320654 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:13.320617    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8" event={"ID":"594bced1-c2e9-45a4-8ec7-edc1c5e0399b","Type":"ContainerStarted","Data":"251daa7aa0469f9582d455fbe4b53b8b3497662ebed01d98378c9778948f1563"}
Jul 28 18:03:13.322092 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:13.322064    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7" event={"ID":"bc2997be-3ec7-4520-b3ec-51477e2e3045","Type":"ContainerStarted","Data":"417e565c64996da2f03de5311cfb74e47733b80eec030378488aad11e63b6867"}
Jul 28 18:03:13.322211 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:13.322105    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7" event={"ID":"bc2997be-3ec7-4520-b3ec-51477e2e3045","Type":"ContainerStarted","Data":"fce23c729946522ce95d7377a01ea569dee2e498c59350f33365b0830e145adc"}
Jul 28 18:03:13.322211 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:13.322128    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"
Jul 28 18:03:13.341150 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:13.341103    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8" podStartSLOduration=1.3410878689999999 podStartE2EDuration="1.341087869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:03:13.339115086 +0000 UTC m=+2789.596724777" watchObservedRunningTime="2026-07-28 18:03:13.341087869 +0000 UTC m=+2789.598697559"
Jul 28 18:03:13.357348 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:13.357284    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7" podStartSLOduration=1.3572629059999999 podStartE2EDuration="1.357262906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:03:13.35687879 +0000 UTC m=+2789.614488481" watchObservedRunningTime="2026-07-28 18:03:13.357262906 +0000 UTC m=+2789.614872598"
Jul 28 18:03:13.511818 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:13.511776    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:22.511557 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:22.511503    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:22.523473 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:22.523439    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:23.377869 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:23.377838    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:03:44.328277 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:44.328245    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"
Jul 28 18:03:54.064146 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:54.064110    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"]
Jul 28 18:03:54.067340 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:54.067320    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"
Jul 28 18:03:54.070292 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:54.070266    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 28 18:03:54.070401 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:54.070271    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-gz99x\""
Jul 28 18:03:54.081844 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:54.081822    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"]
Jul 28 18:03:54.217009 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:54.216977    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnfp7\" (UniqueName: \"kubernetes.io/projected/8b912b0f-c905-4f03-a0c1-c28bec6f0682-kube-api-access-gnfp7\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5\" (UID: \"8b912b0f-c905-4f03-a0c1-c28bec6f0682\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"
Jul 28 18:03:54.217009 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:54.217014    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b912b0f-c905-4f03-a0c1-c28bec6f0682-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5\" (UID: \"8b912b0f-c905-4f03-a0c1-c28bec6f0682\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"
Jul 28 18:03:54.318292 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:54.318216    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b912b0f-c905-4f03-a0c1-c28bec6f0682-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5\" (UID: \"8b912b0f-c905-4f03-a0c1-c28bec6f0682\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"
Jul 28 18:03:54.318455 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:54.318319    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnfp7\" (UniqueName: \"kubernetes.io/projected/8b912b0f-c905-4f03-a0c1-c28bec6f0682-kube-api-access-gnfp7\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5\" (UID: \"8b912b0f-c905-4f03-a0c1-c28bec6f0682\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"
Jul 28 18:03:54.320764 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:54.320734    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b912b0f-c905-4f03-a0c1-c28bec6f0682-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5\" (UID: \"8b912b0f-c905-4f03-a0c1-c28bec6f0682\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"
Jul 28 18:03:54.327354 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:54.327329    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnfp7\" (UniqueName: \"kubernetes.io/projected/8b912b0f-c905-4f03-a0c1-c28bec6f0682-kube-api-access-gnfp7\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5\" (UID: \"8b912b0f-c905-4f03-a0c1-c28bec6f0682\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"
Jul 28 18:03:54.380049 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:54.380016    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"
Jul 28 18:03:54.512105 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:54.511981    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"]
Jul 28 18:03:55.524809 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:55.524773    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5" event={"ID":"8b912b0f-c905-4f03-a0c1-c28bec6f0682","Type":"ContainerStarted","Data":"535d9fae94bf69dc2a0c8524694dd577490a6a0b5e7d0ad0be62a9138e38c889"}
Jul 28 18:03:55.525185 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:55.524824    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5" event={"ID":"8b912b0f-c905-4f03-a0c1-c28bec6f0682","Type":"ContainerStarted","Data":"e4ec23645f18f95af0df002d29f023d4229843aee527a09918d2aa0baba7d899"}
Jul 28 18:03:55.525185 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:55.524881    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"
Jul 28 18:03:55.546269 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:03:55.546218    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5" podStartSLOduration=1.5462048149999998 podStartE2EDuration="1.546204815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:03:55.543934997 +0000 UTC m=+2831.801544686" watchObservedRunningTime="2026-07-28 18:03:55.546204815 +0000 UTC m=+2831.803814504"
Jul 28 18:04:26.532328 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:26.532294    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"
Jul 28 18:04:28.246448 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:28.246391    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"]
Jul 28 18:04:36.533081 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:36.532984    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5" podUID="8b912b0f-c905-4f03-a0c1-c28bec6f0682" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:04:43.275696 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.275623    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5" podUID="8b912b0f-c905-4f03-a0c1-c28bec6f0682" containerName="main" containerID="cri-o://535d9fae94bf69dc2a0c8524694dd577490a6a0b5e7d0ad0be62a9138e38c889" gracePeriod=45
Jul 28 18:04:43.542779 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.542750    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"
Jul 28 18:04:43.673198 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.673153    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnfp7\" (UniqueName: \"kubernetes.io/projected/8b912b0f-c905-4f03-a0c1-c28bec6f0682-kube-api-access-gnfp7\") pod \"8b912b0f-c905-4f03-a0c1-c28bec6f0682\" (UID: \"8b912b0f-c905-4f03-a0c1-c28bec6f0682\") "
Jul 28 18:04:43.673380 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.673243    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b912b0f-c905-4f03-a0c1-c28bec6f0682-tls-certs\") pod \"8b912b0f-c905-4f03-a0c1-c28bec6f0682\" (UID: \"8b912b0f-c905-4f03-a0c1-c28bec6f0682\") "
Jul 28 18:04:43.675684 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.675640    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b912b0f-c905-4f03-a0c1-c28bec6f0682-kube-api-access-gnfp7" (OuterVolumeSpecName: "kube-api-access-gnfp7") pod "8b912b0f-c905-4f03-a0c1-c28bec6f0682" (UID: "8b912b0f-c905-4f03-a0c1-c28bec6f0682"). InnerVolumeSpecName "kube-api-access-gnfp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:04:43.675684 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.675662    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b912b0f-c905-4f03-a0c1-c28bec6f0682-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b912b0f-c905-4f03-a0c1-c28bec6f0682" (UID: "8b912b0f-c905-4f03-a0c1-c28bec6f0682"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:04:43.766202 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.766161    2575 generic.go:356] "Generic (PLEG): container finished" podID="8b912b0f-c905-4f03-a0c1-c28bec6f0682" containerID="535d9fae94bf69dc2a0c8524694dd577490a6a0b5e7d0ad0be62a9138e38c889" exitCode=0
Jul 28 18:04:43.766202 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.766204    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"
Jul 28 18:04:43.766486 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.766248    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5" event={"ID":"8b912b0f-c905-4f03-a0c1-c28bec6f0682","Type":"ContainerDied","Data":"535d9fae94bf69dc2a0c8524694dd577490a6a0b5e7d0ad0be62a9138e38c889"}
Jul 28 18:04:43.766486 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.766306    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5" event={"ID":"8b912b0f-c905-4f03-a0c1-c28bec6f0682","Type":"ContainerDied","Data":"e4ec23645f18f95af0df002d29f023d4229843aee527a09918d2aa0baba7d899"}
Jul 28 18:04:43.766486 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.766348    2575 scope.go:117] "RemoveContainer" containerID="535d9fae94bf69dc2a0c8524694dd577490a6a0b5e7d0ad0be62a9138e38c889"
Jul 28 18:04:43.774344 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.774318    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnfp7\" (UniqueName: \"kubernetes.io/projected/8b912b0f-c905-4f03-a0c1-c28bec6f0682-kube-api-access-gnfp7\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:04:43.774344 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.774342    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b912b0f-c905-4f03-a0c1-c28bec6f0682-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:04:43.776053 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.776034    2575 scope.go:117] "RemoveContainer" containerID="535d9fae94bf69dc2a0c8524694dd577490a6a0b5e7d0ad0be62a9138e38c889"
Jul 28 18:04:43.776364 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:04:43.776345    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535d9fae94bf69dc2a0c8524694dd577490a6a0b5e7d0ad0be62a9138e38c889\": container with ID starting with 535d9fae94bf69dc2a0c8524694dd577490a6a0b5e7d0ad0be62a9138e38c889 not found: ID does not exist" containerID="535d9fae94bf69dc2a0c8524694dd577490a6a0b5e7d0ad0be62a9138e38c889"
Jul 28 18:04:43.776458 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.776373    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535d9fae94bf69dc2a0c8524694dd577490a6a0b5e7d0ad0be62a9138e38c889"} err="failed to get container status \"535d9fae94bf69dc2a0c8524694dd577490a6a0b5e7d0ad0be62a9138e38c889\": rpc error: code = NotFound desc = could not find container \"535d9fae94bf69dc2a0c8524694dd577490a6a0b5e7d0ad0be62a9138e38c889\": container with ID starting with 535d9fae94bf69dc2a0c8524694dd577490a6a0b5e7d0ad0be62a9138e38c889 not found: ID does not exist"
Jul 28 18:04:43.789403 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.789362    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"]
Jul 28 18:04:43.791858 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:43.791826    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb47s5"]
Jul 28 18:04:44.312439 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:44.312405    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b912b0f-c905-4f03-a0c1-c28bec6f0682" path="/var/lib/kubelet/pods/8b912b0f-c905-4f03-a0c1-c28bec6f0682/volumes"
Jul 28 18:04:48.499130 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:48.499090    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"]
Jul 28 18:04:48.508879 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:04:48.508837    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:04:48.508879 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:04:48.508879 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:04:48.508879 ip-10-0-135-188 kubenswrapper[2575]:  > containerID="a4f354ac13c1dc303c1ee779ac0bad802b4f0efaf8054c421277b92b49ebf9d3" cmd=["/bin/sleep","15"]
Jul 28 18:04:48.509362 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:48.508896    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8" message="" err=<
Jul 28 18:04:48.509362 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:04:48.509362 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:04:48.509362 ip-10-0-135-188 kubenswrapper[2575]:  >
Jul 28 18:04:48.509362 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:04:48.508910    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:04:48.509362 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:04:48.509362 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:04:48.509362 ip-10-0-135-188 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8" podUID="594bced1-c2e9-45a4-8ec7-edc1c5e0399b" containerName="main" containerID="cri-o://a4f354ac13c1dc303c1ee779ac0bad802b4f0efaf8054c421277b92b49ebf9d3"
Jul 28 18:04:48.509362 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:48.508961    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8" podUID="594bced1-c2e9-45a4-8ec7-edc1c5e0399b" containerName="main" containerID="cri-o://a4f354ac13c1dc303c1ee779ac0bad802b4f0efaf8054c421277b92b49ebf9d3" gracePeriod=60
Jul 28 18:04:48.511610 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:48.511580    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"]
Jul 28 18:04:48.792934 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:48.792841    2575 generic.go:356] "Generic (PLEG): container finished" podID="594bced1-c2e9-45a4-8ec7-edc1c5e0399b" containerID="a4f354ac13c1dc303c1ee779ac0bad802b4f0efaf8054c421277b92b49ebf9d3" exitCode=0
Jul 28 18:04:48.792934 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:48.792917    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8" event={"ID":"594bced1-c2e9-45a4-8ec7-edc1c5e0399b","Type":"ContainerDied","Data":"a4f354ac13c1dc303c1ee779ac0bad802b4f0efaf8054c421277b92b49ebf9d3"}
Jul 28 18:04:49.269223 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.269193    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:04:49.331181 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.331134    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z548m\" (UniqueName: \"kubernetes.io/projected/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-kube-api-access-z548m\") pod \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") "
Jul 28 18:04:49.331400 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.331204    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-dshm\") pod \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") "
Jul 28 18:04:49.331400 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.331230    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-model-cache\") pod \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") "
Jul 28 18:04:49.331400 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.331288    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-tls-certs\") pod \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") "
Jul 28 18:04:49.331400 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.331309    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-tmp-dir\") pod \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") "
Jul 28 18:04:49.331400 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.331343    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-home\") pod \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\" (UID: \"594bced1-c2e9-45a4-8ec7-edc1c5e0399b\") "
Jul 28 18:04:49.331729 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.331549    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-model-cache" (OuterVolumeSpecName: "model-cache") pod "594bced1-c2e9-45a4-8ec7-edc1c5e0399b" (UID: "594bced1-c2e9-45a4-8ec7-edc1c5e0399b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:49.331729 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.331679    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "594bced1-c2e9-45a4-8ec7-edc1c5e0399b" (UID: "594bced1-c2e9-45a4-8ec7-edc1c5e0399b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:49.331729 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.331710    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-model-cache\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:04:49.331867 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.331749    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-home" (OuterVolumeSpecName: "home") pod "594bced1-c2e9-45a4-8ec7-edc1c5e0399b" (UID: "594bced1-c2e9-45a4-8ec7-edc1c5e0399b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:49.333445 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.333415    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-dshm" (OuterVolumeSpecName: "dshm") pod "594bced1-c2e9-45a4-8ec7-edc1c5e0399b" (UID: "594bced1-c2e9-45a4-8ec7-edc1c5e0399b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:49.333706 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.333684    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-kube-api-access-z548m" (OuterVolumeSpecName: "kube-api-access-z548m") pod "594bced1-c2e9-45a4-8ec7-edc1c5e0399b" (UID: "594bced1-c2e9-45a4-8ec7-edc1c5e0399b"). InnerVolumeSpecName "kube-api-access-z548m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:04:49.333764 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.333723    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "594bced1-c2e9-45a4-8ec7-edc1c5e0399b" (UID: "594bced1-c2e9-45a4-8ec7-edc1c5e0399b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:04:49.432151 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.432117    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:04:49.432151 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.432146    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-tmp-dir\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:04:49.432151 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.432154    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-home\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:04:49.432402 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.432165    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z548m\" (UniqueName: \"kubernetes.io/projected/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-kube-api-access-z548m\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:04:49.432402 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.432174    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/594bced1-c2e9-45a4-8ec7-edc1c5e0399b-dshm\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:04:49.803146 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.803060    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"
Jul 28 18:04:49.803146 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.803094    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8" event={"ID":"594bced1-c2e9-45a4-8ec7-edc1c5e0399b","Type":"ContainerDied","Data":"251daa7aa0469f9582d455fbe4b53b8b3497662ebed01d98378c9778948f1563"}
Jul 28 18:04:49.803666 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.803159    2575 scope.go:117] "RemoveContainer" containerID="a4f354ac13c1dc303c1ee779ac0bad802b4f0efaf8054c421277b92b49ebf9d3"
Jul 28 18:04:49.833782 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.833745    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"]
Jul 28 18:04:49.835383 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:49.835353    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969t8tz8"]
Jul 28 18:04:50.312725 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:50.312688    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="594bced1-c2e9-45a4-8ec7-edc1c5e0399b" path="/var/lib/kubelet/pods/594bced1-c2e9-45a4-8ec7-edc1c5e0399b/volumes"
Jul 28 18:04:54.327933 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:04:54.327886    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7" podUID="bc2997be-3ec7-4520-b3ec-51477e2e3045" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:05:03.531165 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:03.531098    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7" podUID="bc2997be-3ec7-4520-b3ec-51477e2e3045" containerName="main" containerID="cri-o://417e565c64996da2f03de5311cfb74e47733b80eec030378488aad11e63b6867" gracePeriod=45
Jul 28 18:05:03.791750 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:03.791678    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"
Jul 28 18:05:03.869703 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:03.869664    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc2997be-3ec7-4520-b3ec-51477e2e3045-tls-certs\") pod \"bc2997be-3ec7-4520-b3ec-51477e2e3045\" (UID: \"bc2997be-3ec7-4520-b3ec-51477e2e3045\") "
Jul 28 18:05:03.869916 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:03.869786    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsrpn\" (UniqueName: \"kubernetes.io/projected/bc2997be-3ec7-4520-b3ec-51477e2e3045-kube-api-access-rsrpn\") pod \"bc2997be-3ec7-4520-b3ec-51477e2e3045\" (UID: \"bc2997be-3ec7-4520-b3ec-51477e2e3045\") "
Jul 28 18:05:03.871641 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:03.871514    2575 generic.go:356] "Generic (PLEG): container finished" podID="bc2997be-3ec7-4520-b3ec-51477e2e3045" containerID="417e565c64996da2f03de5311cfb74e47733b80eec030378488aad11e63b6867" exitCode=0
Jul 28 18:05:03.871641 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:03.871577    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7" event={"ID":"bc2997be-3ec7-4520-b3ec-51477e2e3045","Type":"ContainerDied","Data":"417e565c64996da2f03de5311cfb74e47733b80eec030378488aad11e63b6867"}
Jul 28 18:05:03.871895 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:03.871601    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"
Jul 28 18:05:03.871895 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:03.871666    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7" event={"ID":"bc2997be-3ec7-4520-b3ec-51477e2e3045","Type":"ContainerDied","Data":"fce23c729946522ce95d7377a01ea569dee2e498c59350f33365b0830e145adc"}
Jul 28 18:05:03.871895 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:03.871702    2575 scope.go:117] "RemoveContainer" containerID="417e565c64996da2f03de5311cfb74e47733b80eec030378488aad11e63b6867"
Jul 28 18:05:03.872281 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:03.872255    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc2997be-3ec7-4520-b3ec-51477e2e3045-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc2997be-3ec7-4520-b3ec-51477e2e3045" (UID: "bc2997be-3ec7-4520-b3ec-51477e2e3045"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:05:03.872379 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:03.872303    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2997be-3ec7-4520-b3ec-51477e2e3045-kube-api-access-rsrpn" (OuterVolumeSpecName: "kube-api-access-rsrpn") pod "bc2997be-3ec7-4520-b3ec-51477e2e3045" (UID: "bc2997be-3ec7-4520-b3ec-51477e2e3045"). InnerVolumeSpecName "kube-api-access-rsrpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:05:03.889539 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:03.889491    2575 scope.go:117] "RemoveContainer" containerID="417e565c64996da2f03de5311cfb74e47733b80eec030378488aad11e63b6867"
Jul 28 18:05:03.889872 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:05:03.889842    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"417e565c64996da2f03de5311cfb74e47733b80eec030378488aad11e63b6867\": container with ID starting with 417e565c64996da2f03de5311cfb74e47733b80eec030378488aad11e63b6867 not found: ID does not exist" containerID="417e565c64996da2f03de5311cfb74e47733b80eec030378488aad11e63b6867"
Jul 28 18:05:03.889938 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:03.889881    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"417e565c64996da2f03de5311cfb74e47733b80eec030378488aad11e63b6867"} err="failed to get container status \"417e565c64996da2f03de5311cfb74e47733b80eec030378488aad11e63b6867\": rpc error: code = NotFound desc = could not find container \"417e565c64996da2f03de5311cfb74e47733b80eec030378488aad11e63b6867\": container with ID starting with 417e565c64996da2f03de5311cfb74e47733b80eec030378488aad11e63b6867 not found: ID does not exist"
Jul 28 18:05:03.971239 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:03.971201    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc2997be-3ec7-4520-b3ec-51477e2e3045-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:05:03.971239 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:03.971236    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsrpn\" (UniqueName: \"kubernetes.io/projected/bc2997be-3ec7-4520-b3ec-51477e2e3045-kube-api-access-rsrpn\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:05:04.217761 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:04.217717    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"]
Jul 28 18:05:04.221155 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:04.221123    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevz9v7"]
Jul 28 18:05:04.312117 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:04.312079    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc2997be-3ec7-4520-b3ec-51477e2e3045" path="/var/lib/kubelet/pods/bc2997be-3ec7-4520-b3ec-51477e2e3045/volumes"
Jul 28 18:05:24.157433 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.157391    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"]
Jul 28 18:05:24.157932 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.157774    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594bced1-c2e9-45a4-8ec7-edc1c5e0399b" containerName="main"
Jul 28 18:05:24.157932 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.157788    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="594bced1-c2e9-45a4-8ec7-edc1c5e0399b" containerName="main"
Jul 28 18:05:24.157932 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.157805    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b912b0f-c905-4f03-a0c1-c28bec6f0682" containerName="main"
Jul 28 18:05:24.157932 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.157811    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b912b0f-c905-4f03-a0c1-c28bec6f0682" containerName="main"
Jul 28 18:05:24.157932 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.157827    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc2997be-3ec7-4520-b3ec-51477e2e3045" containerName="main"
Jul 28 18:05:24.157932 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.157833    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc2997be-3ec7-4520-b3ec-51477e2e3045" containerName="main"
Jul 28 18:05:24.157932 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.157885    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc2997be-3ec7-4520-b3ec-51477e2e3045" containerName="main"
Jul 28 18:05:24.157932 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.157894    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="594bced1-c2e9-45a4-8ec7-edc1c5e0399b" containerName="main"
Jul 28 18:05:24.157932 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.157901    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b912b0f-c905-4f03-a0c1-c28bec6f0682" containerName="main"
Jul 28 18:05:24.162794 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.162773    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"
Jul 28 18:05:24.172213 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.172185    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-bkrf6\""
Jul 28 18:05:24.172375 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.172236    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 28 18:05:24.172465 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.172249    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 28 18:05:24.172571 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.172269    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-xbqqw\""
Jul 28 18:05:24.172644 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.172304    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 28 18:05:24.190747 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.190716    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"]
Jul 28 18:05:24.261247 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.261209    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf\" (UID: \"1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"
Jul 28 18:05:24.261446 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.261289    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn64k\" (UniqueName: \"kubernetes.io/projected/1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020-kube-api-access-dn64k\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf\" (UID: \"1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"
Jul 28 18:05:24.362147 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.362108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf\" (UID: \"1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"
Jul 28 18:05:24.362365 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.362198    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn64k\" (UniqueName: \"kubernetes.io/projected/1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020-kube-api-access-dn64k\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf\" (UID: \"1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"
Jul 28 18:05:24.364860 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.364822    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf\" (UID: \"1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"
Jul 28 18:05:24.378640 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.378611    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn64k\" (UniqueName: \"kubernetes.io/projected/1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020-kube-api-access-dn64k\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf\" (UID: \"1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"
Jul 28 18:05:24.473790 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.473749    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"
Jul 28 18:05:24.636336 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.636303    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"]
Jul 28 18:05:24.639236 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:05:24.639203    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cfeef4f_9cba_42ff_a2a1_3d7e06eb7020.slice/crio-d6d235d9ca307ad3ca88af9960fa17be17524e0061391dea30d9cb8a6fedffc6 WatchSource:0}: Error finding container d6d235d9ca307ad3ca88af9960fa17be17524e0061391dea30d9cb8a6fedffc6: Status 404 returned error can't find the container with id d6d235d9ca307ad3ca88af9960fa17be17524e0061391dea30d9cb8a6fedffc6
Jul 28 18:05:24.972414 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.972369    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf" event={"ID":"1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020","Type":"ContainerStarted","Data":"db49d7db82510af6358dab14269a1f69f2f28fe39a40d8a9964533dade615f74"}
Jul 28 18:05:24.972638 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.972437    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf" event={"ID":"1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020","Type":"ContainerStarted","Data":"d6d235d9ca307ad3ca88af9960fa17be17524e0061391dea30d9cb8a6fedffc6"}
Jul 28 18:05:24.972638 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:24.972496    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"
Jul 28 18:05:25.009845 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:25.009788    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf" podStartSLOduration=1.009770218 podStartE2EDuration="1.009770218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:05:25.008821052 +0000 UTC m=+2921.266430741" watchObservedRunningTime="2026-07-28 18:05:25.009770218 +0000 UTC m=+2921.267379908"
Jul 28 18:05:55.979003 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:05:55.978971    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"
Jul 28 18:06:44.417252 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:06:44.417215    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:06:44.426388 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:06:44.426362    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:07:11.823767 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:11.823728    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"]
Jul 28 18:07:15.978760 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:15.978719    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf" podUID="1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:07:25.978933 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:25.978888    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf" podUID="1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:07:26.843707 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:26.843655    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf" podUID="1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020" containerName="main" containerID="cri-o://db49d7db82510af6358dab14269a1f69f2f28fe39a40d8a9964533dade615f74" gracePeriod=45
Jul 28 18:07:27.113182 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.113150    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"
Jul 28 18:07:27.140149 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.140090    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn64k\" (UniqueName: \"kubernetes.io/projected/1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020-kube-api-access-dn64k\") pod \"1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020\" (UID: \"1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020\") "
Jul 28 18:07:27.140356 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.140185    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020-tls-certs\") pod \"1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020\" (UID: \"1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020\") "
Jul 28 18:07:27.142665 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.142628    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020" (UID: "1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:07:27.142806 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.142730    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020-kube-api-access-dn64k" (OuterVolumeSpecName: "kube-api-access-dn64k") pod "1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020" (UID: "1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020"). InnerVolumeSpecName "kube-api-access-dn64k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:07:27.241390 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.241339    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn64k\" (UniqueName: \"kubernetes.io/projected/1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020-kube-api-access-dn64k\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:07:27.241390 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.241383    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:07:27.567094 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.567052    2575 generic.go:356] "Generic (PLEG): container finished" podID="1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020" containerID="db49d7db82510af6358dab14269a1f69f2f28fe39a40d8a9964533dade615f74" exitCode=0
Jul 28 18:07:27.567094 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.567096    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"
Jul 28 18:07:27.567337 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.567111    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf" event={"ID":"1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020","Type":"ContainerDied","Data":"db49d7db82510af6358dab14269a1f69f2f28fe39a40d8a9964533dade615f74"}
Jul 28 18:07:27.567337 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.567157    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf" event={"ID":"1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020","Type":"ContainerDied","Data":"d6d235d9ca307ad3ca88af9960fa17be17524e0061391dea30d9cb8a6fedffc6"}
Jul 28 18:07:27.567337 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.567182    2575 scope.go:117] "RemoveContainer" containerID="db49d7db82510af6358dab14269a1f69f2f28fe39a40d8a9964533dade615f74"
Jul 28 18:07:27.577021 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.577001    2575 scope.go:117] "RemoveContainer" containerID="db49d7db82510af6358dab14269a1f69f2f28fe39a40d8a9964533dade615f74"
Jul 28 18:07:27.577307 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:07:27.577283    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db49d7db82510af6358dab14269a1f69f2f28fe39a40d8a9964533dade615f74\": container with ID starting with db49d7db82510af6358dab14269a1f69f2f28fe39a40d8a9964533dade615f74 not found: ID does not exist" containerID="db49d7db82510af6358dab14269a1f69f2f28fe39a40d8a9964533dade615f74"
Jul 28 18:07:27.577353 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.577316    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db49d7db82510af6358dab14269a1f69f2f28fe39a40d8a9964533dade615f74"} err="failed to get container status \"db49d7db82510af6358dab14269a1f69f2f28fe39a40d8a9964533dade615f74\": rpc error: code = NotFound desc = could not find container \"db49d7db82510af6358dab14269a1f69f2f28fe39a40d8a9964533dade615f74\": container with ID starting with db49d7db82510af6358dab14269a1f69f2f28fe39a40d8a9964533dade615f74 not found: ID does not exist"
Jul 28 18:07:27.590539 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.590491    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"]
Jul 28 18:07:27.594891 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:27.594853    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche6kljf"]
Jul 28 18:07:28.312579 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:28.312517    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020" path="/var/lib/kubelet/pods/1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020/volumes"
Jul 28 18:07:45.173377 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.173337    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"]
Jul 28 18:07:45.173929 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.173884    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020" containerName="main"
Jul 28 18:07:45.173929 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.173906    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020" containerName="main"
Jul 28 18:07:45.174056 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.174013    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cfeef4f-9cba-42ff-a2a1-3d7e06eb7020" containerName="main"
Jul 28 18:07:45.178441 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.178417    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"
Jul 28 18:07:45.181793 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.181773    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 28 18:07:45.181946 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.181773    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-jljxf\""
Jul 28 18:07:45.181946 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.181773    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 28 18:07:45.181946 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.181813    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 28 18:07:45.181946 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.181831    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-btckm\""
Jul 28 18:07:45.186882 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.186856    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"]
Jul 28 18:07:45.301651 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.301617    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8065f6a9-25d8-45a1-90f1-e0e6ce2eda15-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b\" (UID: \"8065f6a9-25d8-45a1-90f1-e0e6ce2eda15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"
Jul 28 18:07:45.301832 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.301720    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhgxs\" (UniqueName: \"kubernetes.io/projected/8065f6a9-25d8-45a1-90f1-e0e6ce2eda15-kube-api-access-zhgxs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b\" (UID: \"8065f6a9-25d8-45a1-90f1-e0e6ce2eda15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"
Jul 28 18:07:45.403210 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.403170    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhgxs\" (UniqueName: \"kubernetes.io/projected/8065f6a9-25d8-45a1-90f1-e0e6ce2eda15-kube-api-access-zhgxs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b\" (UID: \"8065f6a9-25d8-45a1-90f1-e0e6ce2eda15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"
Jul 28 18:07:45.403384 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.403244    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8065f6a9-25d8-45a1-90f1-e0e6ce2eda15-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b\" (UID: \"8065f6a9-25d8-45a1-90f1-e0e6ce2eda15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"
Jul 28 18:07:45.405956 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.405922    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8065f6a9-25d8-45a1-90f1-e0e6ce2eda15-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b\" (UID: \"8065f6a9-25d8-45a1-90f1-e0e6ce2eda15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"
Jul 28 18:07:45.412026 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.411990    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhgxs\" (UniqueName: \"kubernetes.io/projected/8065f6a9-25d8-45a1-90f1-e0e6ce2eda15-kube-api-access-zhgxs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b\" (UID: \"8065f6a9-25d8-45a1-90f1-e0e6ce2eda15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"
Jul 28 18:07:45.489895 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.489799    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"
Jul 28 18:07:45.635653 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.635538    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"]
Jul 28 18:07:45.638499 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:07:45.638464    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8065f6a9_25d8_45a1_90f1_e0e6ce2eda15.slice/crio-c7a3c7b465b5e136c787a04f617d0f7b990e32f1e0197a60ef94db09d31723a5 WatchSource:0}: Error finding container c7a3c7b465b5e136c787a04f617d0f7b990e32f1e0197a60ef94db09d31723a5: Status 404 returned error can't find the container with id c7a3c7b465b5e136c787a04f617d0f7b990e32f1e0197a60ef94db09d31723a5
Jul 28 18:07:45.656099 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:45.656067    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b" event={"ID":"8065f6a9-25d8-45a1-90f1-e0e6ce2eda15","Type":"ContainerStarted","Data":"c7a3c7b465b5e136c787a04f617d0f7b990e32f1e0197a60ef94db09d31723a5"}
Jul 28 18:07:46.662638 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:46.662582    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b" event={"ID":"8065f6a9-25d8-45a1-90f1-e0e6ce2eda15","Type":"ContainerStarted","Data":"c54415fcc966e0906a618e38ec02a799c40a8824d3fefec94b94e8ec89ef4380"}
Jul 28 18:07:46.663132 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:46.662662    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"
Jul 28 18:07:46.689277 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:07:46.689219    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b" podStartSLOduration=1.6892004250000001 podStartE2EDuration="1.689200425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:07:46.689183947 +0000 UTC m=+3062.946793634" watchObservedRunningTime="2026-07-28 18:07:46.689200425 +0000 UTC m=+3062.946810114"
Jul 28 18:08:17.669385 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:17.669349    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"
Jul 28 18:08:25.904562 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:25.904484    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"]
Jul 28 18:08:25.908123 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:25.908099    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"
Jul 28 18:08:25.910821 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:25.910794    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"openshift-service-ca.crt\""
Jul 28 18:08:25.911117 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:25.911077    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"kube-root-ca.crt\""
Jul 28 18:08:25.911236 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:25.911136    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"default-dockercfg-fnljc\""
Jul 28 18:08:25.911604 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:25.911581    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-epp-sa-dockercfg-shn6d\""
Jul 28 18:08:25.911709 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:25.911679    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-kserve-self-signed-certs\""
Jul 28 18:08:25.917030 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:25.917004    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"]
Jul 28 18:08:25.977451 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:25.977411    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/169c9dd2-a164-4736-8d62-b1e4be44b202-tls-certs\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k\" (UID: \"169c9dd2-a164-4736-8d62-b1e4be44b202\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"
Jul 28 18:08:25.977689 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:25.977563    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72bfm\" (UniqueName: \"kubernetes.io/projected/169c9dd2-a164-4736-8d62-b1e4be44b202-kube-api-access-72bfm\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k\" (UID: \"169c9dd2-a164-4736-8d62-b1e4be44b202\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"
Jul 28 18:08:26.078424 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:26.078384    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bfm\" (UniqueName: \"kubernetes.io/projected/169c9dd2-a164-4736-8d62-b1e4be44b202-kube-api-access-72bfm\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k\" (UID: \"169c9dd2-a164-4736-8d62-b1e4be44b202\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"
Jul 28 18:08:26.078646 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:26.078449    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/169c9dd2-a164-4736-8d62-b1e4be44b202-tls-certs\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k\" (UID: \"169c9dd2-a164-4736-8d62-b1e4be44b202\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"
Jul 28 18:08:26.081283 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:26.081249    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/169c9dd2-a164-4736-8d62-b1e4be44b202-tls-certs\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k\" (UID: \"169c9dd2-a164-4736-8d62-b1e4be44b202\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"
Jul 28 18:08:26.088344 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:26.088309    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bfm\" (UniqueName: \"kubernetes.io/projected/169c9dd2-a164-4736-8d62-b1e4be44b202-kube-api-access-72bfm\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k\" (UID: \"169c9dd2-a164-4736-8d62-b1e4be44b202\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"
Jul 28 18:08:26.221441 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:26.221391    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"
Jul 28 18:08:26.368411 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:26.368365    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"]
Jul 28 18:08:26.371147 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:08:26.371108    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169c9dd2_a164_4736_8d62_b1e4be44b202.slice/crio-b103afb508bb29d728b9d9cfdff37433f94f33588d60307456ed622963efebbb WatchSource:0}: Error finding container b103afb508bb29d728b9d9cfdff37433f94f33588d60307456ed622963efebbb: Status 404 returned error can't find the container with id b103afb508bb29d728b9d9cfdff37433f94f33588d60307456ed622963efebbb
Jul 28 18:08:26.854714 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:26.854671    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k" event={"ID":"169c9dd2-a164-4736-8d62-b1e4be44b202","Type":"ContainerStarted","Data":"9a73e7c941cc97525a3e504fdcef9011abbac8736b7e54b151c26f4d811e5aec"}
Jul 28 18:08:26.854951 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:26.854732    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k" event={"ID":"169c9dd2-a164-4736-8d62-b1e4be44b202","Type":"ContainerStarted","Data":"b103afb508bb29d728b9d9cfdff37433f94f33588d60307456ed622963efebbb"}
Jul 28 18:08:26.854951 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:26.854753    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"
Jul 28 18:08:26.873158 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:26.873099    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k" podStartSLOduration=1.873069809 podStartE2EDuration="1.873069809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:08:26.87217559 +0000 UTC m=+3103.129785281" watchObservedRunningTime="2026-07-28 18:08:26.873069809 +0000 UTC m=+3103.130679499"
Jul 28 18:08:57.861488 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:08:57.861452    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"
Jul 28 18:09:16.856145 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:16.856051    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"]
Jul 28 18:09:17.669500 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:17.669462    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b" podUID="8065f6a9-25d8-45a1-90f1-e0e6ce2eda15" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:09:27.669548 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:27.669480    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b" podUID="8065f6a9-25d8-45a1-90f1-e0e6ce2eda15" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:09:31.877007 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:31.876954    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b" podUID="8065f6a9-25d8-45a1-90f1-e0e6ce2eda15" containerName="main" containerID="cri-o://c54415fcc966e0906a618e38ec02a799c40a8824d3fefec94b94e8ec89ef4380" gracePeriod=45
Jul 28 18:09:32.131345 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.131273    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"
Jul 28 18:09:32.174636 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.174596    2575 generic.go:356] "Generic (PLEG): container finished" podID="8065f6a9-25d8-45a1-90f1-e0e6ce2eda15" containerID="c54415fcc966e0906a618e38ec02a799c40a8824d3fefec94b94e8ec89ef4380" exitCode=0
Jul 28 18:09:32.174841 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.174654    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"
Jul 28 18:09:32.174841 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.174674    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b" event={"ID":"8065f6a9-25d8-45a1-90f1-e0e6ce2eda15","Type":"ContainerDied","Data":"c54415fcc966e0906a618e38ec02a799c40a8824d3fefec94b94e8ec89ef4380"}
Jul 28 18:09:32.174841 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.174731    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b" event={"ID":"8065f6a9-25d8-45a1-90f1-e0e6ce2eda15","Type":"ContainerDied","Data":"c7a3c7b465b5e136c787a04f617d0f7b990e32f1e0197a60ef94db09d31723a5"}
Jul 28 18:09:32.174841 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.174757    2575 scope.go:117] "RemoveContainer" containerID="c54415fcc966e0906a618e38ec02a799c40a8824d3fefec94b94e8ec89ef4380"
Jul 28 18:09:32.184676 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.184651    2575 scope.go:117] "RemoveContainer" containerID="c54415fcc966e0906a618e38ec02a799c40a8824d3fefec94b94e8ec89ef4380"
Jul 28 18:09:32.185000 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:09:32.184981    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c54415fcc966e0906a618e38ec02a799c40a8824d3fefec94b94e8ec89ef4380\": container with ID starting with c54415fcc966e0906a618e38ec02a799c40a8824d3fefec94b94e8ec89ef4380 not found: ID does not exist" containerID="c54415fcc966e0906a618e38ec02a799c40a8824d3fefec94b94e8ec89ef4380"
Jul 28 18:09:32.185048 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.185010    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54415fcc966e0906a618e38ec02a799c40a8824d3fefec94b94e8ec89ef4380"} err="failed to get container status \"c54415fcc966e0906a618e38ec02a799c40a8824d3fefec94b94e8ec89ef4380\": rpc error: code = NotFound desc = could not find container \"c54415fcc966e0906a618e38ec02a799c40a8824d3fefec94b94e8ec89ef4380\": container with ID starting with c54415fcc966e0906a618e38ec02a799c40a8824d3fefec94b94e8ec89ef4380 not found: ID does not exist"
Jul 28 18:09:32.279225 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.279183    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhgxs\" (UniqueName: \"kubernetes.io/projected/8065f6a9-25d8-45a1-90f1-e0e6ce2eda15-kube-api-access-zhgxs\") pod \"8065f6a9-25d8-45a1-90f1-e0e6ce2eda15\" (UID: \"8065f6a9-25d8-45a1-90f1-e0e6ce2eda15\") "
Jul 28 18:09:32.279411 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.279261    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8065f6a9-25d8-45a1-90f1-e0e6ce2eda15-tls-certs\") pod \"8065f6a9-25d8-45a1-90f1-e0e6ce2eda15\" (UID: \"8065f6a9-25d8-45a1-90f1-e0e6ce2eda15\") "
Jul 28 18:09:32.281569 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.281538    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8065f6a9-25d8-45a1-90f1-e0e6ce2eda15-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8065f6a9-25d8-45a1-90f1-e0e6ce2eda15" (UID: "8065f6a9-25d8-45a1-90f1-e0e6ce2eda15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:09:32.281700 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.281587    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8065f6a9-25d8-45a1-90f1-e0e6ce2eda15-kube-api-access-zhgxs" (OuterVolumeSpecName: "kube-api-access-zhgxs") pod "8065f6a9-25d8-45a1-90f1-e0e6ce2eda15" (UID: "8065f6a9-25d8-45a1-90f1-e0e6ce2eda15"). InnerVolumeSpecName "kube-api-access-zhgxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:09:32.380789 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.380744    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhgxs\" (UniqueName: \"kubernetes.io/projected/8065f6a9-25d8-45a1-90f1-e0e6ce2eda15-kube-api-access-zhgxs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:09:32.380789 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.380781    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8065f6a9-25d8-45a1-90f1-e0e6ce2eda15-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:09:32.491297 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.491257    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"]
Jul 28 18:09:32.497286 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:32.497254    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenk56b"]
Jul 28 18:09:34.318601 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:09:34.318564    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8065f6a9-25d8-45a1-90f1-e0e6ce2eda15" path="/var/lib/kubelet/pods/8065f6a9-25d8-45a1-90f1-e0e6ce2eda15/volumes"
Jul 28 18:11:44.444920 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:11:44.444889    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:11:44.456589 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:11:44.456558    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:13:01.352049 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.352011    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"]
Jul 28 18:13:01.352685 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.352409    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8065f6a9-25d8-45a1-90f1-e0e6ce2eda15" containerName="main"
Jul 28 18:13:01.352685 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.352428    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8065f6a9-25d8-45a1-90f1-e0e6ce2eda15" containerName="main"
Jul 28 18:13:01.352685 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.352535    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8065f6a9-25d8-45a1-90f1-e0e6ce2eda15" containerName="main"
Jul 28 18:13:01.355761 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.355737    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.358268 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.358237    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-w6gcf\""
Jul 28 18:13:01.358268 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.358238    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 28 18:13:01.358452 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.358238    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 28 18:13:01.358649 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.358635    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 28 18:13:01.366940 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.366907    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"]
Jul 28 18:13:01.467937 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.467892    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714f9546-db67-453c-b8d0-f91fdaa6a01e-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.467937 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.467936    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714f9546-db67-453c-b8d0-f91fdaa6a01e-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.468220 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.468029    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6ss2\" (UniqueName: \"kubernetes.io/projected/714f9546-db67-453c-b8d0-f91fdaa6a01e-kube-api-access-h6ss2\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.468220 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.468118    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/714f9546-db67-453c-b8d0-f91fdaa6a01e-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.468332 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.468268    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/714f9546-db67-453c-b8d0-f91fdaa6a01e-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.468332 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.468304    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/714f9546-db67-453c-b8d0-f91fdaa6a01e-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.569472 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.569429    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714f9546-db67-453c-b8d0-f91fdaa6a01e-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.569693 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.569481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714f9546-db67-453c-b8d0-f91fdaa6a01e-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.569693 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.569589    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6ss2\" (UniqueName: \"kubernetes.io/projected/714f9546-db67-453c-b8d0-f91fdaa6a01e-kube-api-access-h6ss2\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.569693 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.569648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/714f9546-db67-453c-b8d0-f91fdaa6a01e-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.569693 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.569682    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/714f9546-db67-453c-b8d0-f91fdaa6a01e-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.569931 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.569714    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/714f9546-db67-453c-b8d0-f91fdaa6a01e-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.569988 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.569947    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714f9546-db67-453c-b8d0-f91fdaa6a01e-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.569988 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.569968    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714f9546-db67-453c-b8d0-f91fdaa6a01e-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.570225 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.570204    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/714f9546-db67-453c-b8d0-f91fdaa6a01e-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.572497 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.572468    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/714f9546-db67-453c-b8d0-f91fdaa6a01e-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.572798 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.572775    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/714f9546-db67-453c-b8d0-f91fdaa6a01e-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:01.579080 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:01.579046    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6ss2\" (UniqueName: \"kubernetes.io/projected/714f9546-db67-453c-b8d0-f91fdaa6a01e-kube-api-access-h6ss2\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:05.809962 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:05.809912    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:05.817612 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:05.817581    2575 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:05.911057 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:05.911008    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:05.911244 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:05.911170    2575 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:05.914091 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:05.914053    2575 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") DevicePath \"csi-5bcc2ce21c4b5c028913383f0b9bc17d78d411bd4026f7b723692f4df21d86c1\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:06.012910 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:06.012880    2575 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d80372062c10f11126b77088bee6191fef8e1bbabe19894b05b047257cca869e/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:06.030216 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:06.030180    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-f1df9520-ed9a-482b-9b2b-0f8681809f16\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cc867d06be7d65b\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd\" (UID: \"714f9546-db67-453c-b8d0-f91fdaa6a01e\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:06.167475 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:06.167416    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:06.315748 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:06.315710    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"]
Jul 28 18:13:06.318767 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:13:06.318726    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714f9546_db67_453c_b8d0_f91fdaa6a01e.slice/crio-706a8f48109d08b9b5a6389ad385b5987eed10d30218d66754a61607b3ae4796 WatchSource:0}: Error finding container 706a8f48109d08b9b5a6389ad385b5987eed10d30218d66754a61607b3ae4796: Status 404 returned error can't find the container with id 706a8f48109d08b9b5a6389ad385b5987eed10d30218d66754a61607b3ae4796
Jul 28 18:13:06.321144 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:06.321123    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:13:07.240256 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:07.240214    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd" event={"ID":"714f9546-db67-453c-b8d0-f91fdaa6a01e","Type":"ContainerStarted","Data":"706a8f48109d08b9b5a6389ad385b5987eed10d30218d66754a61607b3ae4796"}
Jul 28 18:13:37.419020 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:37.418972    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd" event={"ID":"714f9546-db67-453c-b8d0-f91fdaa6a01e","Type":"ContainerStarted","Data":"b33819149f394b955aab6820246e62d181de5c977cd84d096a9ecb6caef8a88a"}
Jul 28 18:13:37.441872 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:37.441810    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd" podStartSLOduration=5.675906494 podStartE2EDuration="36.441790022s" totalImagesPullingTime="30.765883528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:13:37.439782054 +0000 UTC m=+3413.697391752" watchObservedRunningTime="2026-07-28 18:13:37.441790022 +0000 UTC m=+3413.699399712"
Jul 28 18:13:46.167847 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:46.167805    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:46.168347 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:46.167904    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:13:46.169741 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:46.169697    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd" podUID="714f9546-db67-453c-b8d0-f91fdaa6a01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 28 18:13:56.168672 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:13:56.168614    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd" podUID="714f9546-db67-453c-b8d0-f91fdaa6a01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 28 18:14:06.168958 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:14:06.168900    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd" podUID="714f9546-db67-453c-b8d0-f91fdaa6a01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 28 18:14:16.168593 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:14:16.168539    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd" podUID="714f9546-db67-453c-b8d0-f91fdaa6a01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 28 18:14:26.168082 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:14:26.168036    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd" podUID="714f9546-db67-453c-b8d0-f91fdaa6a01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 28 18:14:36.168741 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:14:36.168684    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd" podUID="714f9546-db67-453c-b8d0-f91fdaa6a01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 28 18:14:46.167969 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:14:46.167910    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd" podUID="714f9546-db67-453c-b8d0-f91fdaa6a01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 28 18:14:56.168381 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:14:56.168331    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd" podUID="714f9546-db67-453c-b8d0-f91fdaa6a01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 28 18:15:06.168408 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:15:06.168355    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd" podUID="714f9546-db67-453c-b8d0-f91fdaa6a01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 28 18:15:16.168221 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:15:16.168165    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd" podUID="714f9546-db67-453c-b8d0-f91fdaa6a01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 28 18:15:26.168017 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:15:26.167963    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd" podUID="714f9546-db67-453c-b8d0-f91fdaa6a01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 28 18:15:36.168269 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:15:36.168214    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd" podUID="714f9546-db67-453c-b8d0-f91fdaa6a01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 28 18:15:46.179021 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:15:46.178974    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:15:46.187315 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:15:46.187280    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd"
Jul 28 18:16:44.474332 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:16:44.474298    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:16:44.485827 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:16:44.485801    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:21:44.502707 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:21:44.502678    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:21:44.515480 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:21:44.515454    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:23:27.129589 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:27.129544    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"]
Jul 28 18:23:27.861797 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:27.861747    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k" podUID="169c9dd2-a164-4736-8d62-b1e4be44b202" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:23:37.861463 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:37.861418    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k" podUID="169c9dd2-a164-4736-8d62-b1e4be44b202" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:23:42.150043 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:42.149998    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k" podUID="169c9dd2-a164-4736-8d62-b1e4be44b202" containerName="main" containerID="cri-o://9a73e7c941cc97525a3e504fdcef9011abbac8736b7e54b151c26f4d811e5aec" gracePeriod=45
Jul 28 18:23:42.408407 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:42.408321    2575 generic.go:356] "Generic (PLEG): container finished" podID="169c9dd2-a164-4736-8d62-b1e4be44b202" containerID="9a73e7c941cc97525a3e504fdcef9011abbac8736b7e54b151c26f4d811e5aec" exitCode=0
Jul 28 18:23:42.408554 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:42.408392    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k" event={"ID":"169c9dd2-a164-4736-8d62-b1e4be44b202","Type":"ContainerDied","Data":"9a73e7c941cc97525a3e504fdcef9011abbac8736b7e54b151c26f4d811e5aec"}
Jul 28 18:23:42.408554 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:42.408456    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k" event={"ID":"169c9dd2-a164-4736-8d62-b1e4be44b202","Type":"ContainerDied","Data":"b103afb508bb29d728b9d9cfdff37433f94f33588d60307456ed622963efebbb"}
Jul 28 18:23:42.408554 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:42.408475    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b103afb508bb29d728b9d9cfdff37433f94f33588d60307456ed622963efebbb"
Jul 28 18:23:42.413842 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:42.413820    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"
Jul 28 18:23:42.489845 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:42.489802    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/169c9dd2-a164-4736-8d62-b1e4be44b202-tls-certs\") pod \"169c9dd2-a164-4736-8d62-b1e4be44b202\" (UID: \"169c9dd2-a164-4736-8d62-b1e4be44b202\") "
Jul 28 18:23:42.490056 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:42.489866    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72bfm\" (UniqueName: \"kubernetes.io/projected/169c9dd2-a164-4736-8d62-b1e4be44b202-kube-api-access-72bfm\") pod \"169c9dd2-a164-4736-8d62-b1e4be44b202\" (UID: \"169c9dd2-a164-4736-8d62-b1e4be44b202\") "
Jul 28 18:23:42.492259 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:42.492217    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169c9dd2-a164-4736-8d62-b1e4be44b202-kube-api-access-72bfm" (OuterVolumeSpecName: "kube-api-access-72bfm") pod "169c9dd2-a164-4736-8d62-b1e4be44b202" (UID: "169c9dd2-a164-4736-8d62-b1e4be44b202"). InnerVolumeSpecName "kube-api-access-72bfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:23:42.492390 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:42.492266    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/169c9dd2-a164-4736-8d62-b1e4be44b202-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "169c9dd2-a164-4736-8d62-b1e4be44b202" (UID: "169c9dd2-a164-4736-8d62-b1e4be44b202"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:23:42.591095 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:42.591056    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/169c9dd2-a164-4736-8d62-b1e4be44b202-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:23:42.591095 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:42.591092    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72bfm\" (UniqueName: \"kubernetes.io/projected/169c9dd2-a164-4736-8d62-b1e4be44b202-kube-api-access-72bfm\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:23:43.416893 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:43.416849    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"
Jul 28 18:23:43.440469 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:43.440429    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"]
Jul 28 18:23:43.444446 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:43.444405    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-mts9k"]
Jul 28 18:23:44.312875 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:44.312836    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="169c9dd2-a164-4736-8d62-b1e4be44b202" path="/var/lib/kubelet/pods/169c9dd2-a164-4736-8d62-b1e4be44b202/volumes"
Jul 28 18:23:44.733545 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:23:44.733486    2575 scope.go:117] "RemoveContainer" containerID="9a73e7c941cc97525a3e504fdcef9011abbac8736b7e54b151c26f4d811e5aec"
Jul 28 18:24:00.274130 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.274091    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:24:00.274635 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.274460    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169c9dd2-a164-4736-8d62-b1e4be44b202" containerName="main"
Jul 28 18:24:00.274635 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.274473    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="169c9dd2-a164-4736-8d62-b1e4be44b202" containerName="main"
Jul 28 18:24:00.274635 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.274545    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="169c9dd2-a164-4736-8d62-b1e4be44b202" containerName="main"
Jul 28 18:24:00.277207 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.277184    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.280958 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.280932    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-2vkkx\""
Jul 28 18:24:00.281219 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.281196    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-mn-dockercfg-878wk\""
Jul 28 18:24:00.281298 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.281241    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 28 18:24:00.281298 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.281251    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 28 18:24:00.281412 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.281243    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 28 18:24:00.294271 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.294247    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:24:00.340682 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.340645    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"]
Jul 28 18:24:00.342827 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.342806    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"
Jul 28 18:24:00.347175 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.347155    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-epp-sa-dockercfg-tsngz\""
Jul 28 18:24:00.364482 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.364451    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"]
Jul 28 18:24:00.452868 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.452828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.453060 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.452880    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf074acd-3c5f-4154-9952-83c5c617ad4d-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h\" (UID: \"bf074acd-3c5f-4154-9952-83c5c617ad4d\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"
Jul 28 18:24:00.453060 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.452980    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.453181 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.453057    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.453181 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.453095    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0da365-ddd1-43ef-b25b-95bc4db560b3-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.453272 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.453168    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.453333 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.453280    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5s5x\" (UniqueName: \"kubernetes.io/projected/9f0da365-ddd1-43ef-b25b-95bc4db560b3-kube-api-access-w5s5x\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.453333 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.453316    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7nr4\" (UniqueName: \"kubernetes.io/projected/bf074acd-3c5f-4154-9952-83c5c617ad4d-kube-api-access-q7nr4\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h\" (UID: \"bf074acd-3c5f-4154-9952-83c5c617ad4d\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"
Jul 28 18:24:00.554082 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.554000    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.554082 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.554044    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.554082 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.554071    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0da365-ddd1-43ef-b25b-95bc4db560b3-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.554330 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.554097    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.554330 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.554147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5s5x\" (UniqueName: \"kubernetes.io/projected/9f0da365-ddd1-43ef-b25b-95bc4db560b3-kube-api-access-w5s5x\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.554330 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.554172    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nr4\" (UniqueName: \"kubernetes.io/projected/bf074acd-3c5f-4154-9952-83c5c617ad4d-kube-api-access-q7nr4\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h\" (UID: \"bf074acd-3c5f-4154-9952-83c5c617ad4d\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"
Jul 28 18:24:00.554330 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.554223    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.554330 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.554258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf074acd-3c5f-4154-9952-83c5c617ad4d-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h\" (UID: \"bf074acd-3c5f-4154-9952-83c5c617ad4d\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"
Jul 28 18:24:00.554636 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.554393    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.554636 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.554561    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.554709 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.554639    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.556452 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.556420    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.556615 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.556595    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0da365-ddd1-43ef-b25b-95bc4db560b3-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.556779 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.556763    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf074acd-3c5f-4154-9952-83c5c617ad4d-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h\" (UID: \"bf074acd-3c5f-4154-9952-83c5c617ad4d\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"
Jul 28 18:24:00.562244 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.562218    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5s5x\" (UniqueName: \"kubernetes.io/projected/9f0da365-ddd1-43ef-b25b-95bc4db560b3-kube-api-access-w5s5x\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.562513 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.562492    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7nr4\" (UniqueName: \"kubernetes.io/projected/bf074acd-3c5f-4154-9952-83c5c617ad4d-kube-api-access-q7nr4\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h\" (UID: \"bf074acd-3c5f-4154-9952-83c5c617ad4d\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"
Jul 28 18:24:00.588032 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.588013    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:24:00.653484 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.653448    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"
Jul 28 18:24:00.728037 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.728012    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:24:00.730725 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:24:00.730696    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0da365_ddd1_43ef_b25b_95bc4db560b3.slice/crio-681792f3d49dd66e7d7bc831c26fe55f3aedc1f8004d80b93285af3395eed12f WatchSource:0}: Error finding container 681792f3d49dd66e7d7bc831c26fe55f3aedc1f8004d80b93285af3395eed12f: Status 404 returned error can't find the container with id 681792f3d49dd66e7d7bc831c26fe55f3aedc1f8004d80b93285af3395eed12f
Jul 28 18:24:00.795318 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:00.795292    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"]
Jul 28 18:24:00.797494 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:24:00.797469    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf074acd_3c5f_4154_9952_83c5c617ad4d.slice/crio-1ac17f43a0b7c3f1ba13a4d0bb1d97f476aac477efd74e0ae2a1c960b9e11f75 WatchSource:0}: Error finding container 1ac17f43a0b7c3f1ba13a4d0bb1d97f476aac477efd74e0ae2a1c960b9e11f75: Status 404 returned error can't find the container with id 1ac17f43a0b7c3f1ba13a4d0bb1d97f476aac477efd74e0ae2a1c960b9e11f75
Jul 28 18:24:01.501693 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:01.501646    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"9f0da365-ddd1-43ef-b25b-95bc4db560b3","Type":"ContainerStarted","Data":"2e8258862652c4679c3a19c9a1b08ab6a7a960af14434df916d95b2c07e6ea05"}
Jul 28 18:24:01.502180 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:01.501756    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"9f0da365-ddd1-43ef-b25b-95bc4db560b3","Type":"ContainerStarted","Data":"681792f3d49dd66e7d7bc831c26fe55f3aedc1f8004d80b93285af3395eed12f"}
Jul 28 18:24:01.503076 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:01.503050    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h" event={"ID":"bf074acd-3c5f-4154-9952-83c5c617ad4d","Type":"ContainerStarted","Data":"cbb90abd9abd69896dd068badd58f8cb3425f2dd1b75b8a340548e1bf38120b9"}
Jul 28 18:24:01.503156 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:01.503098    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h" event={"ID":"bf074acd-3c5f-4154-9952-83c5c617ad4d","Type":"ContainerStarted","Data":"1ac17f43a0b7c3f1ba13a4d0bb1d97f476aac477efd74e0ae2a1c960b9e11f75"}
Jul 28 18:24:01.503156 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:01.503136    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"
Jul 28 18:24:01.519352 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:01.519302    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podStartSLOduration=1.519289769 podStartE2EDuration="1.519289769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:24:01.518730677 +0000 UTC m=+4037.776340359" watchObservedRunningTime="2026-07-28 18:24:01.519289769 +0000 UTC m=+4037.776899459"
Jul 28 18:24:01.535002 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:01.534941    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h" podStartSLOduration=1.534921004 podStartE2EDuration="1.534921004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:24:01.533391933 +0000 UTC m=+4037.791001623" watchObservedRunningTime="2026-07-28 18:24:01.534921004 +0000 UTC m=+4037.792530695"
Jul 28 18:24:32.509697 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:24:32.509621    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"
Jul 28 18:26:44.531621 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:26:44.531481    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:26:44.545786 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:26:44.545767    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:27:48.421427 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:27:48.421392    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bwg5gd_714f9546-db67-453c-b8d0-f91fdaa6a01e/main/0.log"
Jul 28 18:28:47.325990 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.325898    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"]
Jul 28 18:28:47.328498 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.328476    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"
Jul 28 18:28:47.331589 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.331567    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 28 18:28:47.332463 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.332415    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-vzspf\""
Jul 28 18:28:47.332463 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.332448    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 28 18:28:47.332691 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.332486    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 28 18:28:47.332933 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.332919    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-hwprp\""
Jul 28 18:28:47.356276 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.356229    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"]
Jul 28 18:28:47.436119 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.436082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd16acc-a694-43f0-9070-24a879fab4ee-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4\" (UID: \"6bd16acc-a694-43f0-9070-24a879fab4ee\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"
Jul 28 18:28:47.436119 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.436131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2vt9\" (UniqueName: \"kubernetes.io/projected/6bd16acc-a694-43f0-9070-24a879fab4ee-kube-api-access-v2vt9\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4\" (UID: \"6bd16acc-a694-43f0-9070-24a879fab4ee\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"
Jul 28 18:28:47.536658 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.536626    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd16acc-a694-43f0-9070-24a879fab4ee-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4\" (UID: \"6bd16acc-a694-43f0-9070-24a879fab4ee\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"
Jul 28 18:28:47.536658 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.536661    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vt9\" (UniqueName: \"kubernetes.io/projected/6bd16acc-a694-43f0-9070-24a879fab4ee-kube-api-access-v2vt9\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4\" (UID: \"6bd16acc-a694-43f0-9070-24a879fab4ee\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"
Jul 28 18:28:47.539249 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.539224    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd16acc-a694-43f0-9070-24a879fab4ee-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4\" (UID: \"6bd16acc-a694-43f0-9070-24a879fab4ee\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"
Jul 28 18:28:47.545935 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.545907    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2vt9\" (UniqueName: \"kubernetes.io/projected/6bd16acc-a694-43f0-9070-24a879fab4ee-kube-api-access-v2vt9\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4\" (UID: \"6bd16acc-a694-43f0-9070-24a879fab4ee\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"
Jul 28 18:28:47.638909 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.638827    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"
Jul 28 18:28:47.780088 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.780060    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"]
Jul 28 18:28:47.782760 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:28:47.782727    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd16acc_a694_43f0_9070_24a879fab4ee.slice/crio-b117832456f07f44e1f51ff07e2c9deadb826566e67f0e9a543bcb19c699bda9 WatchSource:0}: Error finding container b117832456f07f44e1f51ff07e2c9deadb826566e67f0e9a543bcb19c699bda9: Status 404 returned error can't find the container with id b117832456f07f44e1f51ff07e2c9deadb826566e67f0e9a543bcb19c699bda9
Jul 28 18:28:47.850140 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.850112    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4" event={"ID":"6bd16acc-a694-43f0-9070-24a879fab4ee","Type":"ContainerStarted","Data":"976229a7fb89a7569e5e7883cceac8e479185bf593b7d809690a772aa62b4ee7"}
Jul 28 18:28:47.850261 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.850187    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4" event={"ID":"6bd16acc-a694-43f0-9070-24a879fab4ee","Type":"ContainerStarted","Data":"b117832456f07f44e1f51ff07e2c9deadb826566e67f0e9a543bcb19c699bda9"}
Jul 28 18:28:47.850261 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.850215    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"
Jul 28 18:28:47.869120 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:28:47.869074    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4" podStartSLOduration=0.869058931 podStartE2EDuration="869.058931ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:28:47.868914106 +0000 UTC m=+4324.126523796" watchObservedRunningTime="2026-07-28 18:28:47.869058931 +0000 UTC m=+4324.126668618"
Jul 28 18:29:18.857775 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:29:18.857743    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"
Jul 28 18:30:42.879751 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:42.879707    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"]
Jul 28 18:30:48.857654 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:48.857602    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4" podUID="6bd16acc-a694-43f0-9070-24a879fab4ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:30:57.901934 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:57.901885    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4" podUID="6bd16acc-a694-43f0-9070-24a879fab4ee" containerName="main" containerID="cri-o://976229a7fb89a7569e5e7883cceac8e479185bf593b7d809690a772aa62b4ee7" gracePeriod=45
Jul 28 18:30:58.175783 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.175755    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"
Jul 28 18:30:58.280169 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.280133    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd16acc-a694-43f0-9070-24a879fab4ee-tls-certs\") pod \"6bd16acc-a694-43f0-9070-24a879fab4ee\" (UID: \"6bd16acc-a694-43f0-9070-24a879fab4ee\") "
Jul 28 18:30:58.280364 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.280224    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2vt9\" (UniqueName: \"kubernetes.io/projected/6bd16acc-a694-43f0-9070-24a879fab4ee-kube-api-access-v2vt9\") pod \"6bd16acc-a694-43f0-9070-24a879fab4ee\" (UID: \"6bd16acc-a694-43f0-9070-24a879fab4ee\") "
Jul 28 18:30:58.282476 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.282436    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd16acc-a694-43f0-9070-24a879fab4ee-kube-api-access-v2vt9" (OuterVolumeSpecName: "kube-api-access-v2vt9") pod "6bd16acc-a694-43f0-9070-24a879fab4ee" (UID: "6bd16acc-a694-43f0-9070-24a879fab4ee"). InnerVolumeSpecName "kube-api-access-v2vt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:30:58.282613 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.282472    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd16acc-a694-43f0-9070-24a879fab4ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6bd16acc-a694-43f0-9070-24a879fab4ee" (UID: "6bd16acc-a694-43f0-9070-24a879fab4ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:30:58.381531 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.381463    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd16acc-a694-43f0-9070-24a879fab4ee-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:30:58.381531 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.381506    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2vt9\" (UniqueName: \"kubernetes.io/projected/6bd16acc-a694-43f0-9070-24a879fab4ee-kube-api-access-v2vt9\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:30:58.483627 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.483585    2575 generic.go:356] "Generic (PLEG): container finished" podID="6bd16acc-a694-43f0-9070-24a879fab4ee" containerID="976229a7fb89a7569e5e7883cceac8e479185bf593b7d809690a772aa62b4ee7" exitCode=0
Jul 28 18:30:58.483820 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.483633    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"
Jul 28 18:30:58.483820 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.483669    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4" event={"ID":"6bd16acc-a694-43f0-9070-24a879fab4ee","Type":"ContainerDied","Data":"976229a7fb89a7569e5e7883cceac8e479185bf593b7d809690a772aa62b4ee7"}
Jul 28 18:30:58.483820 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.483723    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4" event={"ID":"6bd16acc-a694-43f0-9070-24a879fab4ee","Type":"ContainerDied","Data":"b117832456f07f44e1f51ff07e2c9deadb826566e67f0e9a543bcb19c699bda9"}
Jul 28 18:30:58.483820 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.483752    2575 scope.go:117] "RemoveContainer" containerID="976229a7fb89a7569e5e7883cceac8e479185bf593b7d809690a772aa62b4ee7"
Jul 28 18:30:58.492672 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.492649    2575 scope.go:117] "RemoveContainer" containerID="976229a7fb89a7569e5e7883cceac8e479185bf593b7d809690a772aa62b4ee7"
Jul 28 18:30:58.492946 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:30:58.492923    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976229a7fb89a7569e5e7883cceac8e479185bf593b7d809690a772aa62b4ee7\": container with ID starting with 976229a7fb89a7569e5e7883cceac8e479185bf593b7d809690a772aa62b4ee7 not found: ID does not exist" containerID="976229a7fb89a7569e5e7883cceac8e479185bf593b7d809690a772aa62b4ee7"
Jul 28 18:30:58.493018 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.492955    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976229a7fb89a7569e5e7883cceac8e479185bf593b7d809690a772aa62b4ee7"} err="failed to get container status \"976229a7fb89a7569e5e7883cceac8e479185bf593b7d809690a772aa62b4ee7\": rpc error: code = NotFound desc = could not find container \"976229a7fb89a7569e5e7883cceac8e479185bf593b7d809690a772aa62b4ee7\": container with ID starting with 976229a7fb89a7569e5e7883cceac8e479185bf593b7d809690a772aa62b4ee7 not found: ID does not exist"
Jul 28 18:30:58.505689 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.505650    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"]
Jul 28 18:30:58.507362 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:30:58.507333    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche4hmh4"]
Jul 28 18:31:00.312536 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:00.312489    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bd16acc-a694-43f0-9070-24a879fab4ee" path="/var/lib/kubelet/pods/6bd16acc-a694-43f0-9070-24a879fab4ee/volumes"
Jul 28 18:31:44.560111 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:44.559993    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:31:44.575334 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:44.575298    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:31:48.050831 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.050787    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"]
Jul 28 18:31:48.051413 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.051319    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd16acc-a694-43f0-9070-24a879fab4ee" containerName="main"
Jul 28 18:31:48.051413 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.051342    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd16acc-a694-43f0-9070-24a879fab4ee" containerName="main"
Jul 28 18:31:48.051500 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.051447    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd16acc-a694-43f0-9070-24a879fab4ee" containerName="main"
Jul 28 18:31:48.054448 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.054424    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"
Jul 28 18:31:48.057711 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.057685    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 28 18:31:48.057911 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.057755    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-vtm2f\""
Jul 28 18:31:48.057911 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.057800    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 28 18:31:48.057911 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.057842    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 28 18:31:48.057911 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.057755    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-cvgb4\""
Jul 28 18:31:48.063590 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.063560    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"]
Jul 28 18:31:48.157312 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.157264    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983e0605-7a70-43ee-a14e-1e99630fa707-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br\" (UID: \"983e0605-7a70-43ee-a14e-1e99630fa707\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"
Jul 28 18:31:48.157502 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.157408    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qssd6\" (UniqueName: \"kubernetes.io/projected/983e0605-7a70-43ee-a14e-1e99630fa707-kube-api-access-qssd6\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br\" (UID: \"983e0605-7a70-43ee-a14e-1e99630fa707\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"
Jul 28 18:31:48.258489 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.258442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qssd6\" (UniqueName: \"kubernetes.io/projected/983e0605-7a70-43ee-a14e-1e99630fa707-kube-api-access-qssd6\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br\" (UID: \"983e0605-7a70-43ee-a14e-1e99630fa707\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"
Jul 28 18:31:48.258724 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.258542    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983e0605-7a70-43ee-a14e-1e99630fa707-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br\" (UID: \"983e0605-7a70-43ee-a14e-1e99630fa707\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"
Jul 28 18:31:48.261499 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.261467    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983e0605-7a70-43ee-a14e-1e99630fa707-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br\" (UID: \"983e0605-7a70-43ee-a14e-1e99630fa707\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"
Jul 28 18:31:48.270195 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.270157    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qssd6\" (UniqueName: \"kubernetes.io/projected/983e0605-7a70-43ee-a14e-1e99630fa707-kube-api-access-qssd6\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br\" (UID: \"983e0605-7a70-43ee-a14e-1e99630fa707\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"
Jul 28 18:31:48.367135 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.367020    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"
Jul 28 18:31:48.514858 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.514828    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"]
Jul 28 18:31:48.516872 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:31:48.516840    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983e0605_7a70_43ee_a14e_1e99630fa707.slice/crio-260b8c94b0da26097965f83730b1d0d884e8cea59a683dcede863e4f88183e26 WatchSource:0}: Error finding container 260b8c94b0da26097965f83730b1d0d884e8cea59a683dcede863e4f88183e26: Status 404 returned error can't find the container with id 260b8c94b0da26097965f83730b1d0d884e8cea59a683dcede863e4f88183e26
Jul 28 18:31:48.730915 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.730876    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br" event={"ID":"983e0605-7a70-43ee-a14e-1e99630fa707","Type":"ContainerStarted","Data":"5ebf609a8121ffa688d6334f120af051d96d15e509e76cb339755f8532fc5ee9"}
Jul 28 18:31:48.731115 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.730936    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br" event={"ID":"983e0605-7a70-43ee-a14e-1e99630fa707","Type":"ContainerStarted","Data":"260b8c94b0da26097965f83730b1d0d884e8cea59a683dcede863e4f88183e26"}
Jul 28 18:31:48.731115 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.731014    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"
Jul 28 18:31:48.750067 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:31:48.750008    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br" podStartSLOduration=0.749987417 podStartE2EDuration="749.987417ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:31:48.747684756 +0000 UTC m=+4505.005294446" watchObservedRunningTime="2026-07-28 18:31:48.749987417 +0000 UTC m=+4505.007597131"
Jul 28 18:32:19.738959 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:32:19.738926    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"
Jul 28 18:34:05.551668 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:05.551630    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"]
Jul 28 18:34:09.738011 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:09.737970    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br" podUID="983e0605-7a70-43ee-a14e-1e99630fa707" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:34:19.738540 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:19.738485    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br" podUID="983e0605-7a70-43ee-a14e-1e99630fa707" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:34:20.571688 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:20.571623    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br" podUID="983e0605-7a70-43ee-a14e-1e99630fa707" containerName="main" containerID="cri-o://5ebf609a8121ffa688d6334f120af051d96d15e509e76cb339755f8532fc5ee9" gracePeriod=45
Jul 28 18:34:20.833784 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:20.833710    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"
Jul 28 18:34:20.993321 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:20.993272    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qssd6\" (UniqueName: \"kubernetes.io/projected/983e0605-7a70-43ee-a14e-1e99630fa707-kube-api-access-qssd6\") pod \"983e0605-7a70-43ee-a14e-1e99630fa707\" (UID: \"983e0605-7a70-43ee-a14e-1e99630fa707\") "
Jul 28 18:34:20.993510 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:20.993355    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983e0605-7a70-43ee-a14e-1e99630fa707-tls-certs\") pod \"983e0605-7a70-43ee-a14e-1e99630fa707\" (UID: \"983e0605-7a70-43ee-a14e-1e99630fa707\") "
Jul 28 18:34:20.995721 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:20.995683    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983e0605-7a70-43ee-a14e-1e99630fa707-kube-api-access-qssd6" (OuterVolumeSpecName: "kube-api-access-qssd6") pod "983e0605-7a70-43ee-a14e-1e99630fa707" (UID: "983e0605-7a70-43ee-a14e-1e99630fa707"). InnerVolumeSpecName "kube-api-access-qssd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:34:20.995843 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:20.995765    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/983e0605-7a70-43ee-a14e-1e99630fa707-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "983e0605-7a70-43ee-a14e-1e99630fa707" (UID: "983e0605-7a70-43ee-a14e-1e99630fa707"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:34:21.094865 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:21.094763    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983e0605-7a70-43ee-a14e-1e99630fa707-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:34:21.094865 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:21.094799    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qssd6\" (UniqueName: \"kubernetes.io/projected/983e0605-7a70-43ee-a14e-1e99630fa707-kube-api-access-qssd6\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:34:21.504922 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:21.504879    2575 generic.go:356] "Generic (PLEG): container finished" podID="983e0605-7a70-43ee-a14e-1e99630fa707" containerID="5ebf609a8121ffa688d6334f120af051d96d15e509e76cb339755f8532fc5ee9" exitCode=0
Jul 28 18:34:21.505227 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:21.504926    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"
Jul 28 18:34:21.505227 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:21.504968    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br" event={"ID":"983e0605-7a70-43ee-a14e-1e99630fa707","Type":"ContainerDied","Data":"5ebf609a8121ffa688d6334f120af051d96d15e509e76cb339755f8532fc5ee9"}
Jul 28 18:34:21.505227 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:21.505030    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br" event={"ID":"983e0605-7a70-43ee-a14e-1e99630fa707","Type":"ContainerDied","Data":"260b8c94b0da26097965f83730b1d0d884e8cea59a683dcede863e4f88183e26"}
Jul 28 18:34:21.505227 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:21.505061    2575 scope.go:117] "RemoveContainer" containerID="5ebf609a8121ffa688d6334f120af051d96d15e509e76cb339755f8532fc5ee9"
Jul 28 18:34:21.515184 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:21.515156    2575 scope.go:117] "RemoveContainer" containerID="5ebf609a8121ffa688d6334f120af051d96d15e509e76cb339755f8532fc5ee9"
Jul 28 18:34:21.515501 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:34:21.515479    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ebf609a8121ffa688d6334f120af051d96d15e509e76cb339755f8532fc5ee9\": container with ID starting with 5ebf609a8121ffa688d6334f120af051d96d15e509e76cb339755f8532fc5ee9 not found: ID does not exist" containerID="5ebf609a8121ffa688d6334f120af051d96d15e509e76cb339755f8532fc5ee9"
Jul 28 18:34:21.515583 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:21.515510    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebf609a8121ffa688d6334f120af051d96d15e509e76cb339755f8532fc5ee9"} err="failed to get container status \"5ebf609a8121ffa688d6334f120af051d96d15e509e76cb339755f8532fc5ee9\": rpc error: code = NotFound desc = could not find container \"5ebf609a8121ffa688d6334f120af051d96d15e509e76cb339755f8532fc5ee9\": container with ID starting with 5ebf609a8121ffa688d6334f120af051d96d15e509e76cb339755f8532fc5ee9 not found: ID does not exist"
Jul 28 18:34:21.529360 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:21.528873    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"]
Jul 28 18:34:21.539673 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:21.539633    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-f82br"]
Jul 28 18:34:22.312933 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:22.312893    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="983e0605-7a70-43ee-a14e-1e99630fa707" path="/var/lib/kubelet/pods/983e0605-7a70-43ee-a14e-1e99630fa707/volumes"
Jul 28 18:34:47.762808 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.762765    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"]
Jul 28 18:34:47.763443 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.763421    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="983e0605-7a70-43ee-a14e-1e99630fa707" containerName="main"
Jul 28 18:34:47.763584 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.763446    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="983e0605-7a70-43ee-a14e-1e99630fa707" containerName="main"
Jul 28 18:34:47.763584 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.763550    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="983e0605-7a70-43ee-a14e-1e99630fa707" containerName="main"
Jul 28 18:34:47.767857 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.767829    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"
Jul 28 18:34:47.770607 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.770570    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 28 18:34:47.770740 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.770631    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-45248\""
Jul 28 18:34:47.771037 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.770980    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-vtm2f\""
Jul 28 18:34:47.771132 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.771034    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 28 18:34:47.771191 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.771136    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 28 18:34:47.779281 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.779250    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"]
Jul 28 18:34:47.853074 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.853021    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8668db3-89c4-4e91-9b44-3097e41bb35f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc\" (UID: \"a8668db3-89c4-4e91-9b44-3097e41bb35f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"
Jul 28 18:34:47.853277 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.853138    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr68r\" (UniqueName: \"kubernetes.io/projected/a8668db3-89c4-4e91-9b44-3097e41bb35f-kube-api-access-dr68r\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc\" (UID: \"a8668db3-89c4-4e91-9b44-3097e41bb35f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"
Jul 28 18:34:47.954335 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.954291    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8668db3-89c4-4e91-9b44-3097e41bb35f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc\" (UID: \"a8668db3-89c4-4e91-9b44-3097e41bb35f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"
Jul 28 18:34:47.954567 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.954367    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr68r\" (UniqueName: \"kubernetes.io/projected/a8668db3-89c4-4e91-9b44-3097e41bb35f-kube-api-access-dr68r\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc\" (UID: \"a8668db3-89c4-4e91-9b44-3097e41bb35f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"
Jul 28 18:34:47.957173 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.957135    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8668db3-89c4-4e91-9b44-3097e41bb35f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc\" (UID: \"a8668db3-89c4-4e91-9b44-3097e41bb35f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"
Jul 28 18:34:47.962027 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:47.961993    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr68r\" (UniqueName: \"kubernetes.io/projected/a8668db3-89c4-4e91-9b44-3097e41bb35f-kube-api-access-dr68r\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc\" (UID: \"a8668db3-89c4-4e91-9b44-3097e41bb35f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"
Jul 28 18:34:48.081823 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:48.081722    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"
Jul 28 18:34:48.225988 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:48.225952    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"]
Jul 28 18:34:48.228341 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:34:48.228302    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8668db3_89c4_4e91_9b44_3097e41bb35f.slice/crio-24874d14ed683b67a484c03b48a4413d00b619a04878fd8003088b484e3ca849 WatchSource:0}: Error finding container 24874d14ed683b67a484c03b48a4413d00b619a04878fd8003088b484e3ca849: Status 404 returned error can't find the container with id 24874d14ed683b67a484c03b48a4413d00b619a04878fd8003088b484e3ca849
Jul 28 18:34:48.649724 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:48.649671    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc" event={"ID":"a8668db3-89c4-4e91-9b44-3097e41bb35f","Type":"ContainerStarted","Data":"ee5b37faa9f7ae351f5de82b0473769adba73deb61e7ac6651d1f8a21747cc27"}
Jul 28 18:34:48.649724 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:48.649732    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc" event={"ID":"a8668db3-89c4-4e91-9b44-3097e41bb35f","Type":"ContainerStarted","Data":"24874d14ed683b67a484c03b48a4413d00b619a04878fd8003088b484e3ca849"}
Jul 28 18:34:48.649997 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:48.649777    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"
Jul 28 18:34:48.668333 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:34:48.668273    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc" podStartSLOduration=1.668255699 podStartE2EDuration="1.668255699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:34:48.666213994 +0000 UTC m=+4684.923823685" watchObservedRunningTime="2026-07-28 18:34:48.668255699 +0000 UTC m=+4684.925865388"
Jul 28 18:35:19.656844 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:35:19.656807    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"
Jul 28 18:36:44.588738 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:36:44.588613    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:36:44.606166 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:36:44.606138    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:36:59.265061 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:36:59.265019    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"]
Jul 28 18:36:59.657248 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:36:59.657151    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc" podUID="a8668db3-89c4-4e91-9b44-3097e41bb35f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:37:09.656279 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:09.656229    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc" podUID="a8668db3-89c4-4e91-9b44-3097e41bb35f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:37:14.285918 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:14.285873    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc" podUID="a8668db3-89c4-4e91-9b44-3097e41bb35f" containerName="main" containerID="cri-o://ee5b37faa9f7ae351f5de82b0473769adba73deb61e7ac6651d1f8a21747cc27" gracePeriod=45
Jul 28 18:37:14.549233 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:14.549147    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"
Jul 28 18:37:14.663826 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:14.663779    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8668db3-89c4-4e91-9b44-3097e41bb35f-tls-certs\") pod \"a8668db3-89c4-4e91-9b44-3097e41bb35f\" (UID: \"a8668db3-89c4-4e91-9b44-3097e41bb35f\") "
Jul 28 18:37:14.664093 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:14.663865    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr68r\" (UniqueName: \"kubernetes.io/projected/a8668db3-89c4-4e91-9b44-3097e41bb35f-kube-api-access-dr68r\") pod \"a8668db3-89c4-4e91-9b44-3097e41bb35f\" (UID: \"a8668db3-89c4-4e91-9b44-3097e41bb35f\") "
Jul 28 18:37:14.666207 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:14.666167    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8668db3-89c4-4e91-9b44-3097e41bb35f-kube-api-access-dr68r" (OuterVolumeSpecName: "kube-api-access-dr68r") pod "a8668db3-89c4-4e91-9b44-3097e41bb35f" (UID: "a8668db3-89c4-4e91-9b44-3097e41bb35f"). InnerVolumeSpecName "kube-api-access-dr68r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:37:14.666207 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:14.666190    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8668db3-89c4-4e91-9b44-3097e41bb35f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8668db3-89c4-4e91-9b44-3097e41bb35f" (UID: "a8668db3-89c4-4e91-9b44-3097e41bb35f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:37:14.764967 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:14.764920    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8668db3-89c4-4e91-9b44-3097e41bb35f-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:37:14.764967 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:14.764962    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr68r\" (UniqueName: \"kubernetes.io/projected/a8668db3-89c4-4e91-9b44-3097e41bb35f-kube-api-access-dr68r\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:37:15.375684 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:15.375644    2575 generic.go:356] "Generic (PLEG): container finished" podID="a8668db3-89c4-4e91-9b44-3097e41bb35f" containerID="ee5b37faa9f7ae351f5de82b0473769adba73deb61e7ac6651d1f8a21747cc27" exitCode=0
Jul 28 18:37:15.375684 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:15.375690    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc" event={"ID":"a8668db3-89c4-4e91-9b44-3097e41bb35f","Type":"ContainerDied","Data":"ee5b37faa9f7ae351f5de82b0473769adba73deb61e7ac6651d1f8a21747cc27"}
Jul 28 18:37:15.376281 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:15.375700    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"
Jul 28 18:37:15.376281 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:15.375735    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc" event={"ID":"a8668db3-89c4-4e91-9b44-3097e41bb35f","Type":"ContainerDied","Data":"24874d14ed683b67a484c03b48a4413d00b619a04878fd8003088b484e3ca849"}
Jul 28 18:37:15.376281 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:15.375757    2575 scope.go:117] "RemoveContainer" containerID="ee5b37faa9f7ae351f5de82b0473769adba73deb61e7ac6651d1f8a21747cc27"
Jul 28 18:37:15.387713 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:15.387683    2575 scope.go:117] "RemoveContainer" containerID="ee5b37faa9f7ae351f5de82b0473769adba73deb61e7ac6651d1f8a21747cc27"
Jul 28 18:37:15.388089 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:37:15.388067    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee5b37faa9f7ae351f5de82b0473769adba73deb61e7ac6651d1f8a21747cc27\": container with ID starting with ee5b37faa9f7ae351f5de82b0473769adba73deb61e7ac6651d1f8a21747cc27 not found: ID does not exist" containerID="ee5b37faa9f7ae351f5de82b0473769adba73deb61e7ac6651d1f8a21747cc27"
Jul 28 18:37:15.388163 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:15.388096    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5b37faa9f7ae351f5de82b0473769adba73deb61e7ac6651d1f8a21747cc27"} err="failed to get container status \"ee5b37faa9f7ae351f5de82b0473769adba73deb61e7ac6651d1f8a21747cc27\": rpc error: code = NotFound desc = could not find container \"ee5b37faa9f7ae351f5de82b0473769adba73deb61e7ac6651d1f8a21747cc27\": container with ID starting with ee5b37faa9f7ae351f5de82b0473769adba73deb61e7ac6651d1f8a21747cc27 not found: ID does not exist"
Jul 28 18:37:15.399751 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:15.399713    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"]
Jul 28 18:37:15.403105 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:15.403071    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-gt2xc"]
Jul 28 18:37:16.312669 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:37:16.312619    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8668db3-89c4-4e91-9b44-3097e41bb35f" path="/var/lib/kubelet/pods/a8668db3-89c4-4e91-9b44-3097e41bb35f/volumes"
Jul 28 18:39:13.875675 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:13.875632    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"]
Jul 28 18:39:14.767143 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:14.767096    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:39:14.776857 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:39:14.776813    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:39:14.776857 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:39:14.776857 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:39:14.776857 ip-10-0-135-188 kubenswrapper[2575]:  > containerID="2e8258862652c4679c3a19c9a1b08ab6a7a960af14434df916d95b2c07e6ea05" cmd=["/bin/sleep","15"]
Jul 28 18:39:14.777123 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:14.776881    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" message="" err=<
Jul 28 18:39:14.777123 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:39:14.777123 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:39:14.777123 ip-10-0-135-188 kubenswrapper[2575]:  >
Jul 28 18:39:14.777123 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:39:14.776893    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:39:14.777123 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:39:14.777123 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:39:14.777123 ip-10-0-135-188 kubenswrapper[2575]:  > pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="9f0da365-ddd1-43ef-b25b-95bc4db560b3" containerName="main" containerID="cri-o://2e8258862652c4679c3a19c9a1b08ab6a7a960af14434df916d95b2c07e6ea05"
Jul 28 18:39:14.777123 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:14.776945    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="9f0da365-ddd1-43ef-b25b-95bc4db560b3" containerName="main" containerID="cri-o://2e8258862652c4679c3a19c9a1b08ab6a7a960af14434df916d95b2c07e6ea05" gracePeriod=60
Jul 28 18:39:14.984230 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:14.984189    2575 generic.go:356] "Generic (PLEG): container finished" podID="9f0da365-ddd1-43ef-b25b-95bc4db560b3" containerID="2e8258862652c4679c3a19c9a1b08ab6a7a960af14434df916d95b2c07e6ea05" exitCode=0
Jul 28 18:39:14.984659 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:14.984253    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"9f0da365-ddd1-43ef-b25b-95bc4db560b3","Type":"ContainerDied","Data":"2e8258862652c4679c3a19c9a1b08ab6a7a960af14434df916d95b2c07e6ea05"}
Jul 28 18:39:15.038196 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.038165    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:39:15.191803 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.191749    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-dshm\") pod \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") "
Jul 28 18:39:15.192009 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.191855    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-home\") pod \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") "
Jul 28 18:39:15.192009 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.191875    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-model-cache\") pod \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") "
Jul 28 18:39:15.192009 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.191915    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0da365-ddd1-43ef-b25b-95bc4db560b3-tls-certs\") pod \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") "
Jul 28 18:39:15.192009 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.191945    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-tmp-dir\") pod \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") "
Jul 28 18:39:15.192009 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.191981    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5s5x\" (UniqueName: \"kubernetes.io/projected/9f0da365-ddd1-43ef-b25b-95bc4db560b3-kube-api-access-w5s5x\") pod \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\" (UID: \"9f0da365-ddd1-43ef-b25b-95bc4db560b3\") "
Jul 28 18:39:15.192286 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.192151    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f0da365-ddd1-43ef-b25b-95bc4db560b3" (UID: "9f0da365-ddd1-43ef-b25b-95bc4db560b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:39:15.192286 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.192186    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-home" (OuterVolumeSpecName: "home") pod "9f0da365-ddd1-43ef-b25b-95bc4db560b3" (UID: "9f0da365-ddd1-43ef-b25b-95bc4db560b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:39:15.192286 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.192259    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f0da365-ddd1-43ef-b25b-95bc4db560b3" (UID: "9f0da365-ddd1-43ef-b25b-95bc4db560b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:39:15.192443 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.192362    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-tmp-dir\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:39:15.192443 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.192380    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-home\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:39:15.192443 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.192392    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-model-cache\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:39:15.194218 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.194190    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-dshm" (OuterVolumeSpecName: "dshm") pod "9f0da365-ddd1-43ef-b25b-95bc4db560b3" (UID: "9f0da365-ddd1-43ef-b25b-95bc4db560b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:39:15.194333 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.194262    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0da365-ddd1-43ef-b25b-95bc4db560b3-kube-api-access-w5s5x" (OuterVolumeSpecName: "kube-api-access-w5s5x") pod "9f0da365-ddd1-43ef-b25b-95bc4db560b3" (UID: "9f0da365-ddd1-43ef-b25b-95bc4db560b3"). InnerVolumeSpecName "kube-api-access-w5s5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:39:15.194333 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.194297    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f0da365-ddd1-43ef-b25b-95bc4db560b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f0da365-ddd1-43ef-b25b-95bc4db560b3" (UID: "9f0da365-ddd1-43ef-b25b-95bc4db560b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:39:15.293642 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.293517    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0da365-ddd1-43ef-b25b-95bc4db560b3-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:39:15.293642 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.293580    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5s5x\" (UniqueName: \"kubernetes.io/projected/9f0da365-ddd1-43ef-b25b-95bc4db560b3-kube-api-access-w5s5x\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:39:15.293642 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.293592    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0da365-ddd1-43ef-b25b-95bc4db560b3-dshm\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:39:15.991056 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.991009    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"9f0da365-ddd1-43ef-b25b-95bc4db560b3","Type":"ContainerDied","Data":"681792f3d49dd66e7d7bc831c26fe55f3aedc1f8004d80b93285af3395eed12f"}
Jul 28 18:39:15.991587 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.991063    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:39:15.991587 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:15.991108    2575 scope.go:117] "RemoveContainer" containerID="2e8258862652c4679c3a19c9a1b08ab6a7a960af14434df916d95b2c07e6ea05"
Jul 28 18:39:16.014860 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:16.014819    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:39:16.020434 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:16.020402    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:39:16.312389 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:16.312296    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f0da365-ddd1-43ef-b25b-95bc4db560b3" path="/var/lib/kubelet/pods/9f0da365-ddd1-43ef-b25b-95bc4db560b3/volumes"
Jul 28 18:39:22.509628 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:22.509575    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h" podUID="bf074acd-3c5f-4154-9952-83c5c617ad4d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:39:28.896238 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:28.896189    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h" podUID="bf074acd-3c5f-4154-9952-83c5c617ad4d" containerName="main" containerID="cri-o://cbb90abd9abd69896dd068badd58f8cb3425f2dd1b75b8a340548e1bf38120b9" gracePeriod=45
Jul 28 18:39:29.058610 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:29.058573    2575 generic.go:356] "Generic (PLEG): container finished" podID="bf074acd-3c5f-4154-9952-83c5c617ad4d" containerID="cbb90abd9abd69896dd068badd58f8cb3425f2dd1b75b8a340548e1bf38120b9" exitCode=0
Jul 28 18:39:29.058785 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:29.058649    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h" event={"ID":"bf074acd-3c5f-4154-9952-83c5c617ad4d","Type":"ContainerDied","Data":"cbb90abd9abd69896dd068badd58f8cb3425f2dd1b75b8a340548e1bf38120b9"}
Jul 28 18:39:29.167908 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:29.167877    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"
Jul 28 18:39:29.225836 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:29.225794    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf074acd-3c5f-4154-9952-83c5c617ad4d-tls-certs\") pod \"bf074acd-3c5f-4154-9952-83c5c617ad4d\" (UID: \"bf074acd-3c5f-4154-9952-83c5c617ad4d\") "
Jul 28 18:39:29.225836 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:29.225839    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7nr4\" (UniqueName: \"kubernetes.io/projected/bf074acd-3c5f-4154-9952-83c5c617ad4d-kube-api-access-q7nr4\") pod \"bf074acd-3c5f-4154-9952-83c5c617ad4d\" (UID: \"bf074acd-3c5f-4154-9952-83c5c617ad4d\") "
Jul 28 18:39:29.228060 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:29.228024    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf074acd-3c5f-4154-9952-83c5c617ad4d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf074acd-3c5f-4154-9952-83c5c617ad4d" (UID: "bf074acd-3c5f-4154-9952-83c5c617ad4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:39:29.228190 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:29.228099    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf074acd-3c5f-4154-9952-83c5c617ad4d-kube-api-access-q7nr4" (OuterVolumeSpecName: "kube-api-access-q7nr4") pod "bf074acd-3c5f-4154-9952-83c5c617ad4d" (UID: "bf074acd-3c5f-4154-9952-83c5c617ad4d"). InnerVolumeSpecName "kube-api-access-q7nr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:39:29.326918 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:29.326869    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7nr4\" (UniqueName: \"kubernetes.io/projected/bf074acd-3c5f-4154-9952-83c5c617ad4d-kube-api-access-q7nr4\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:39:29.326918 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:29.326909    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf074acd-3c5f-4154-9952-83c5c617ad4d-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:39:30.064627 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:30.064586    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h" event={"ID":"bf074acd-3c5f-4154-9952-83c5c617ad4d","Type":"ContainerDied","Data":"1ac17f43a0b7c3f1ba13a4d0bb1d97f476aac477efd74e0ae2a1c960b9e11f75"}
Jul 28 18:39:30.065156 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:30.064631    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"
Jul 28 18:39:30.065156 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:30.064658    2575 scope.go:117] "RemoveContainer" containerID="cbb90abd9abd69896dd068badd58f8cb3425f2dd1b75b8a340548e1bf38120b9"
Jul 28 18:39:30.093112 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:30.090027    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"]
Jul 28 18:39:30.096053 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:30.096016    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-sr86h"]
Jul 28 18:39:30.312344 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:39:30.312291    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf074acd-3c5f-4154-9952-83c5c617ad4d" path="/var/lib/kubelet/pods/bf074acd-3c5f-4154-9952-83c5c617ad4d/volumes"
Jul 28 18:40:01.589489 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.589437    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 18:40:01.590179 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.590002    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0da365-ddd1-43ef-b25b-95bc4db560b3" containerName="main"
Jul 28 18:40:01.590179 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.590022    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0da365-ddd1-43ef-b25b-95bc4db560b3" containerName="main"
Jul 28 18:40:01.590179 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.590036    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf074acd-3c5f-4154-9952-83c5c617ad4d" containerName="main"
Jul 28 18:40:01.590179 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.590047    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf074acd-3c5f-4154-9952-83c5c617ad4d" containerName="main"
Jul 28 18:40:01.590179 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.590064    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8668db3-89c4-4e91-9b44-3097e41bb35f" containerName="main"
Jul 28 18:40:01.590179 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.590079    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8668db3-89c4-4e91-9b44-3097e41bb35f" containerName="main"
Jul 28 18:40:01.590179 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.590169    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8668db3-89c4-4e91-9b44-3097e41bb35f" containerName="main"
Jul 28 18:40:01.590179 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.590179    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f0da365-ddd1-43ef-b25b-95bc4db560b3" containerName="main"
Jul 28 18:40:01.590445 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.590187    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf074acd-3c5f-4154-9952-83c5c617ad4d" containerName="main"
Jul 28 18:40:01.594825 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.594797    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.597990 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.597955    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\""
Jul 28 18:40:01.598152 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.598016    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\""
Jul 28 18:40:01.598152 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.598027    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-6f6mj\""
Jul 28 18:40:01.598152 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.598100    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\""
Jul 28 18:40:01.598152 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.598142    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-mn-dockercfg-5m9w7\""
Jul 28 18:40:01.603147 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.603014    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 18:40:01.722889 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.722839    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.722889 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.722890    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.723125 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.722917    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.723125 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.722979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.723125 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.722995    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsrkf\" (UniqueName: \"kubernetes.io/projected/611db002-424a-4342-a932-c1a4f31b189a-kube-api-access-hsrkf\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.723125 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.723045    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/611db002-424a-4342-a932-c1a4f31b189a-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.780768 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.780729    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"]
Jul 28 18:40:01.784134 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.784111    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"
Jul 28 18:40:01.787268 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.787245    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-epp-sa-dockercfg-2bz9j\""
Jul 28 18:40:01.797170 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.797135    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"]
Jul 28 18:40:01.823537 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.823484    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/611db002-424a-4342-a932-c1a4f31b189a-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.823741 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.823588    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.823741 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.823630    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.823863 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.823763    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.823863 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.823852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.823969 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.823876    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsrkf\" (UniqueName: \"kubernetes.io/projected/611db002-424a-4342-a932-c1a4f31b189a-kube-api-access-hsrkf\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.824026 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.823979    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.824131 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.824103    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.824215 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.824181    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.826067 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.826043    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.826186 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.826095    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/611db002-424a-4342-a932-c1a4f31b189a-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.838746 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.838705    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsrkf\" (UniqueName: \"kubernetes.io/projected/611db002-424a-4342-a932-c1a4f31b189a-kube-api-access-hsrkf\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.906696 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.906581    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:40:01.925450 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.925409    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rz7\" (UniqueName: \"kubernetes.io/projected/25adce7a-e90e-41ac-9b24-865bc444cae1-kube-api-access-p5rz7\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr\" (UID: \"25adce7a-e90e-41ac-9b24-865bc444cae1\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"
Jul 28 18:40:01.925691 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:01.925495    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25adce7a-e90e-41ac-9b24-865bc444cae1-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr\" (UID: \"25adce7a-e90e-41ac-9b24-865bc444cae1\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"
Jul 28 18:40:02.026085 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:02.026039    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25adce7a-e90e-41ac-9b24-865bc444cae1-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr\" (UID: \"25adce7a-e90e-41ac-9b24-865bc444cae1\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"
Jul 28 18:40:02.026270 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:02.026144    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rz7\" (UniqueName: \"kubernetes.io/projected/25adce7a-e90e-41ac-9b24-865bc444cae1-kube-api-access-p5rz7\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr\" (UID: \"25adce7a-e90e-41ac-9b24-865bc444cae1\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"
Jul 28 18:40:02.028900 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:02.028866    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25adce7a-e90e-41ac-9b24-865bc444cae1-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr\" (UID: \"25adce7a-e90e-41ac-9b24-865bc444cae1\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"
Jul 28 18:40:02.037904 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:02.037870    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rz7\" (UniqueName: \"kubernetes.io/projected/25adce7a-e90e-41ac-9b24-865bc444cae1-kube-api-access-p5rz7\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr\" (UID: \"25adce7a-e90e-41ac-9b24-865bc444cae1\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"
Jul 28 18:40:02.070156 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:02.069837    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 18:40:02.073359 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:40:02.073317    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611db002_424a_4342_a932_c1a4f31b189a.slice/crio-c1c0999cb2c5b2a65567dd8d12f779e27214fe4e803b1b956d0a97a8f9ba7700 WatchSource:0}: Error finding container c1c0999cb2c5b2a65567dd8d12f779e27214fe4e803b1b956d0a97a8f9ba7700: Status 404 returned error can't find the container with id c1c0999cb2c5b2a65567dd8d12f779e27214fe4e803b1b956d0a97a8f9ba7700
Jul 28 18:40:02.095151 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:02.095121    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"
Jul 28 18:40:02.221314 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:02.221271    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"611db002-424a-4342-a932-c1a4f31b189a","Type":"ContainerStarted","Data":"0e5a2530f7e93540f44970d5242d47d9426e8615db67139c41c8d726121f46ce"}
Jul 28 18:40:02.221507 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:02.221339    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"611db002-424a-4342-a932-c1a4f31b189a","Type":"ContainerStarted","Data":"c1c0999cb2c5b2a65567dd8d12f779e27214fe4e803b1b956d0a97a8f9ba7700"}
Jul 28 18:40:02.244708 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:02.244641    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podStartSLOduration=1.244621967 podStartE2EDuration="1.244621967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:40:02.243894141 +0000 UTC m=+4998.501503832" watchObservedRunningTime="2026-07-28 18:40:02.244621967 +0000 UTC m=+4998.502231657"
Jul 28 18:40:02.262788 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:02.262747    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"]
Jul 28 18:40:02.266240 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:40:02.266207    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25adce7a_e90e_41ac_9b24_865bc444cae1.slice/crio-efef086e82afffe5737f3300939627097a8ea0c576bb343bb9a0ae896ec54c6c WatchSource:0}: Error finding container efef086e82afffe5737f3300939627097a8ea0c576bb343bb9a0ae896ec54c6c: Status 404 returned error can't find the container with id efef086e82afffe5737f3300939627097a8ea0c576bb343bb9a0ae896ec54c6c
Jul 28 18:40:03.227839 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:03.227798    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr" event={"ID":"25adce7a-e90e-41ac-9b24-865bc444cae1","Type":"ContainerStarted","Data":"c55a9fae774c93cf6b3283fcf02e2f85d2b9ca5edd0ff8ee54b7524c0e9996fa"}
Jul 28 18:40:03.228248 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:03.227860    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr" event={"ID":"25adce7a-e90e-41ac-9b24-865bc444cae1","Type":"ContainerStarted","Data":"efef086e82afffe5737f3300939627097a8ea0c576bb343bb9a0ae896ec54c6c"}
Jul 28 18:40:03.247164 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:03.247097    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr" podStartSLOduration=2.247080392 podStartE2EDuration="2.247080392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:40:03.245116335 +0000 UTC m=+4999.502726026" watchObservedRunningTime="2026-07-28 18:40:03.247080392 +0000 UTC m=+4999.504690082"
Jul 28 18:40:04.232837 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:04.232798    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"
Jul 28 18:40:35.240533 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:40:35.240488    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"
Jul 28 18:41:44.617744 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:41:44.617606    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:41:44.637662 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:41:44.637624    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:43:35.186001 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.185965    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"]
Jul 28 18:43:35.189727 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.189697    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"
Jul 28 18:43:35.192937 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.192880    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 28 18:43:35.192937 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.192879    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-bgmt6\""
Jul 28 18:43:35.192937 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.192889    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 28 18:43:35.193203 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.192880    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-nnc8d\""
Jul 28 18:43:35.193203 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.193170    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 28 18:43:35.199175 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.199142    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"]
Jul 28 18:43:35.335961 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.335913    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/238efdb2-14cb-4e9a-8ae5-b8988a74ae49-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-lg7fd\" (UID: \"238efdb2-14cb-4e9a-8ae5-b8988a74ae49\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"
Jul 28 18:43:35.335961 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.335963    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsxm8\" (UniqueName: \"kubernetes.io/projected/238efdb2-14cb-4e9a-8ae5-b8988a74ae49-kube-api-access-vsxm8\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-lg7fd\" (UID: \"238efdb2-14cb-4e9a-8ae5-b8988a74ae49\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"
Jul 28 18:43:35.437362 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.437251    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/238efdb2-14cb-4e9a-8ae5-b8988a74ae49-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-lg7fd\" (UID: \"238efdb2-14cb-4e9a-8ae5-b8988a74ae49\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"
Jul 28 18:43:35.437362 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.437311    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsxm8\" (UniqueName: \"kubernetes.io/projected/238efdb2-14cb-4e9a-8ae5-b8988a74ae49-kube-api-access-vsxm8\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-lg7fd\" (UID: \"238efdb2-14cb-4e9a-8ae5-b8988a74ae49\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"
Jul 28 18:43:35.440214 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.440175    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/238efdb2-14cb-4e9a-8ae5-b8988a74ae49-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-lg7fd\" (UID: \"238efdb2-14cb-4e9a-8ae5-b8988a74ae49\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"
Jul 28 18:43:35.446186 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.446147    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsxm8\" (UniqueName: \"kubernetes.io/projected/238efdb2-14cb-4e9a-8ae5-b8988a74ae49-kube-api-access-vsxm8\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-lg7fd\" (UID: \"238efdb2-14cb-4e9a-8ae5-b8988a74ae49\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"
Jul 28 18:43:35.502972 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.502921    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"
Jul 28 18:43:35.657224 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:35.657174    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"]
Jul 28 18:43:35.660206 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:43:35.660172    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238efdb2_14cb_4e9a_8ae5_b8988a74ae49.slice/crio-4a5004f2db79ae94689429d79d1e6e3df48790957868bc0ce67b65afcd2952f3 WatchSource:0}: Error finding container 4a5004f2db79ae94689429d79d1e6e3df48790957868bc0ce67b65afcd2952f3: Status 404 returned error can't find the container with id 4a5004f2db79ae94689429d79d1e6e3df48790957868bc0ce67b65afcd2952f3
Jul 28 18:43:36.280218 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:36.280176    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd" event={"ID":"238efdb2-14cb-4e9a-8ae5-b8988a74ae49","Type":"ContainerStarted","Data":"4d17c0e7b32739d38bf210fbdb84bbee81dec68852647500b1e2475697d27f50"}
Jul 28 18:43:36.280654 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:36.280238    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd" event={"ID":"238efdb2-14cb-4e9a-8ae5-b8988a74ae49","Type":"ContainerStarted","Data":"4a5004f2db79ae94689429d79d1e6e3df48790957868bc0ce67b65afcd2952f3"}
Jul 28 18:43:36.280654 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:36.280266    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"
Jul 28 18:43:36.299917 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:43:36.299856    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd" podStartSLOduration=1.299838525 podStartE2EDuration="1.299838525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:43:36.297843693 +0000 UTC m=+5212.555453384" watchObservedRunningTime="2026-07-28 18:43:36.299838525 +0000 UTC m=+5212.557448214"
Jul 28 18:44:07.287053 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:44:07.287014    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"
Jul 28 18:45:42.115921 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:42.115877    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"]
Jul 28 18:45:47.287624 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:47.287581    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd" podUID="238efdb2-14cb-4e9a-8ae5-b8988a74ae49" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:45:57.140018 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:57.139965    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd" podUID="238efdb2-14cb-4e9a-8ae5-b8988a74ae49" containerName="main" containerID="cri-o://4d17c0e7b32739d38bf210fbdb84bbee81dec68852647500b1e2475697d27f50" gracePeriod=45
Jul 28 18:45:57.286743 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:45:57.286708    2575 logging.go:55] [core] [Channel #4979 SubChannel #4980]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.69:9003", ServerName: "10.132.0.69:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.69:9003: connect: connection refused"
Jul 28 18:45:57.406732 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:57.406700    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"
Jul 28 18:45:57.472330 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:57.472288    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsxm8\" (UniqueName: \"kubernetes.io/projected/238efdb2-14cb-4e9a-8ae5-b8988a74ae49-kube-api-access-vsxm8\") pod \"238efdb2-14cb-4e9a-8ae5-b8988a74ae49\" (UID: \"238efdb2-14cb-4e9a-8ae5-b8988a74ae49\") "
Jul 28 18:45:57.472508 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:57.472361    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/238efdb2-14cb-4e9a-8ae5-b8988a74ae49-tls-certs\") pod \"238efdb2-14cb-4e9a-8ae5-b8988a74ae49\" (UID: \"238efdb2-14cb-4e9a-8ae5-b8988a74ae49\") "
Jul 28 18:45:57.474666 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:57.474629    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/238efdb2-14cb-4e9a-8ae5-b8988a74ae49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "238efdb2-14cb-4e9a-8ae5-b8988a74ae49" (UID: "238efdb2-14cb-4e9a-8ae5-b8988a74ae49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:45:57.474666 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:57.474646    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/238efdb2-14cb-4e9a-8ae5-b8988a74ae49-kube-api-access-vsxm8" (OuterVolumeSpecName: "kube-api-access-vsxm8") pod "238efdb2-14cb-4e9a-8ae5-b8988a74ae49" (UID: "238efdb2-14cb-4e9a-8ae5-b8988a74ae49"). InnerVolumeSpecName "kube-api-access-vsxm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:45:57.573766 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:57.573723    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/238efdb2-14cb-4e9a-8ae5-b8988a74ae49-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:45:57.573766 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:57.573759    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsxm8\" (UniqueName: \"kubernetes.io/projected/238efdb2-14cb-4e9a-8ae5-b8988a74ae49-kube-api-access-vsxm8\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:45:58.012823 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:58.012781    2575 generic.go:356] "Generic (PLEG): container finished" podID="238efdb2-14cb-4e9a-8ae5-b8988a74ae49" containerID="4d17c0e7b32739d38bf210fbdb84bbee81dec68852647500b1e2475697d27f50" exitCode=0
Jul 28 18:45:58.013032 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:58.012827    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"
Jul 28 18:45:58.013032 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:58.012859    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd" event={"ID":"238efdb2-14cb-4e9a-8ae5-b8988a74ae49","Type":"ContainerDied","Data":"4d17c0e7b32739d38bf210fbdb84bbee81dec68852647500b1e2475697d27f50"}
Jul 28 18:45:58.013032 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:58.012913    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd" event={"ID":"238efdb2-14cb-4e9a-8ae5-b8988a74ae49","Type":"ContainerDied","Data":"4a5004f2db79ae94689429d79d1e6e3df48790957868bc0ce67b65afcd2952f3"}
Jul 28 18:45:58.013032 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:58.012935    2575 scope.go:117] "RemoveContainer" containerID="4d17c0e7b32739d38bf210fbdb84bbee81dec68852647500b1e2475697d27f50"
Jul 28 18:45:58.022887 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:58.022865    2575 scope.go:117] "RemoveContainer" containerID="4d17c0e7b32739d38bf210fbdb84bbee81dec68852647500b1e2475697d27f50"
Jul 28 18:45:58.023238 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:45:58.023218    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d17c0e7b32739d38bf210fbdb84bbee81dec68852647500b1e2475697d27f50\": container with ID starting with 4d17c0e7b32739d38bf210fbdb84bbee81dec68852647500b1e2475697d27f50 not found: ID does not exist" containerID="4d17c0e7b32739d38bf210fbdb84bbee81dec68852647500b1e2475697d27f50"
Jul 28 18:45:58.023290 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:58.023249    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d17c0e7b32739d38bf210fbdb84bbee81dec68852647500b1e2475697d27f50"} err="failed to get container status \"4d17c0e7b32739d38bf210fbdb84bbee81dec68852647500b1e2475697d27f50\": rpc error: code = NotFound desc = could not find container \"4d17c0e7b32739d38bf210fbdb84bbee81dec68852647500b1e2475697d27f50\": container with ID starting with 4d17c0e7b32739d38bf210fbdb84bbee81dec68852647500b1e2475697d27f50 not found: ID does not exist"
Jul 28 18:45:58.036549 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:58.036497    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"]
Jul 28 18:45:58.040176 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:58.040144    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd"]
Jul 28 18:45:58.287244 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:58.287134    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-lg7fd" podUID="238efdb2-14cb-4e9a-8ae5-b8988a74ae49" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.69:9003\" within 1s: context deadline exceeded"
Jul 28 18:45:58.287244 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:45:58.287187    2575 logging.go:55] [core] [Channel #4979 SubChannel #4980]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.69:9003", ServerName: "10.132.0.69:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.69:9003: operation was canceled"
Jul 28 18:45:58.313182 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:45:58.313147    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="238efdb2-14cb-4e9a-8ae5-b8988a74ae49" path="/var/lib/kubelet/pods/238efdb2-14cb-4e9a-8ae5-b8988a74ae49/volumes"
Jul 28 18:46:18.533357 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.533311    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"]
Jul 28 18:46:18.533845 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.533806    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="238efdb2-14cb-4e9a-8ae5-b8988a74ae49" containerName="main"
Jul 28 18:46:18.533845 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.533828    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="238efdb2-14cb-4e9a-8ae5-b8988a74ae49" containerName="main"
Jul 28 18:46:18.534011 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.533929    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="238efdb2-14cb-4e9a-8ae5-b8988a74ae49" containerName="main"
Jul 28 18:46:18.536918 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.536897    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"
Jul 28 18:46:18.540032 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.540010    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 28 18:46:18.540160 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.540107    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 28 18:46:18.540160 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.540114    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-brxdv\""
Jul 28 18:46:18.540286 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.540158    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 28 18:46:18.540286 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.540225    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-g4489\""
Jul 28 18:46:18.546222 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.546191    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"]
Jul 28 18:46:18.577146 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.577099    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh9fx\" (UniqueName: \"kubernetes.io/projected/9451573d-26ec-4478-af0e-7ad34326e92f-kube-api-access-dh9fx\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8\" (UID: \"9451573d-26ec-4478-af0e-7ad34326e92f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"
Jul 28 18:46:18.577341 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.577260    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9451573d-26ec-4478-af0e-7ad34326e92f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8\" (UID: \"9451573d-26ec-4478-af0e-7ad34326e92f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"
Jul 28 18:46:18.678405 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.678357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9fx\" (UniqueName: \"kubernetes.io/projected/9451573d-26ec-4478-af0e-7ad34326e92f-kube-api-access-dh9fx\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8\" (UID: \"9451573d-26ec-4478-af0e-7ad34326e92f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"
Jul 28 18:46:18.678626 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.678470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9451573d-26ec-4478-af0e-7ad34326e92f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8\" (UID: \"9451573d-26ec-4478-af0e-7ad34326e92f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"
Jul 28 18:46:18.681150 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.681121    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9451573d-26ec-4478-af0e-7ad34326e92f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8\" (UID: \"9451573d-26ec-4478-af0e-7ad34326e92f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"
Jul 28 18:46:18.687072 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.687036    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh9fx\" (UniqueName: \"kubernetes.io/projected/9451573d-26ec-4478-af0e-7ad34326e92f-kube-api-access-dh9fx\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8\" (UID: \"9451573d-26ec-4478-af0e-7ad34326e92f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"
Jul 28 18:46:18.849301 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.849198    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"
Jul 28 18:46:18.997276 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:18.997243    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"]
Jul 28 18:46:18.999399 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:46:18.999354    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9451573d_26ec_4478_af0e_7ad34326e92f.slice/crio-0d7a849980fecc9cdbaab2c18095ac495dcdee1eeb846d5c04bffe30a3c0e6b1 WatchSource:0}: Error finding container 0d7a849980fecc9cdbaab2c18095ac495dcdee1eeb846d5c04bffe30a3c0e6b1: Status 404 returned error can't find the container with id 0d7a849980fecc9cdbaab2c18095ac495dcdee1eeb846d5c04bffe30a3c0e6b1
Jul 28 18:46:19.128219 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:19.128104    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8" event={"ID":"9451573d-26ec-4478-af0e-7ad34326e92f","Type":"ContainerStarted","Data":"1a201dab0e7a539a15cbedd66318e2d864f3a1d1207a7b9c1b02ec4f1c1229b1"}
Jul 28 18:46:19.128219 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:19.128179    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8" event={"ID":"9451573d-26ec-4478-af0e-7ad34326e92f","Type":"ContainerStarted","Data":"0d7a849980fecc9cdbaab2c18095ac495dcdee1eeb846d5c04bffe30a3c0e6b1"}
Jul 28 18:46:19.128426 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:19.128259    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"
Jul 28 18:46:19.145685 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:19.145613    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8" podStartSLOduration=1.145590314 podStartE2EDuration="1.145590314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:46:19.144766452 +0000 UTC m=+5375.402376143" watchObservedRunningTime="2026-07-28 18:46:19.145590314 +0000 UTC m=+5375.403200007"
Jul 28 18:46:44.651777 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:44.651659    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:46:44.676298 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:44.676267    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:46:50.136295 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:46:50.136254    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"
Jul 28 18:49:50.501973 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:49:50.501868    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"]
Jul 28 18:50:00.135921 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:00.135869    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8" podUID="9451573d-26ec-4478-af0e-7ad34326e92f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:50:05.523178 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:05.523128    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8" podUID="9451573d-26ec-4478-af0e-7ad34326e92f" containerName="main" containerID="cri-o://1a201dab0e7a539a15cbedd66318e2d864f3a1d1207a7b9c1b02ec4f1c1229b1" gracePeriod=45
Jul 28 18:50:05.791579 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:05.791545    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"
Jul 28 18:50:05.931896 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:05.931852    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9451573d-26ec-4478-af0e-7ad34326e92f-tls-certs\") pod \"9451573d-26ec-4478-af0e-7ad34326e92f\" (UID: \"9451573d-26ec-4478-af0e-7ad34326e92f\") "
Jul 28 18:50:05.932117 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:05.931940    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dh9fx\" (UniqueName: \"kubernetes.io/projected/9451573d-26ec-4478-af0e-7ad34326e92f-kube-api-access-dh9fx\") pod \"9451573d-26ec-4478-af0e-7ad34326e92f\" (UID: \"9451573d-26ec-4478-af0e-7ad34326e92f\") "
Jul 28 18:50:05.934454 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:05.934412    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9451573d-26ec-4478-af0e-7ad34326e92f-kube-api-access-dh9fx" (OuterVolumeSpecName: "kube-api-access-dh9fx") pod "9451573d-26ec-4478-af0e-7ad34326e92f" (UID: "9451573d-26ec-4478-af0e-7ad34326e92f"). InnerVolumeSpecName "kube-api-access-dh9fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:50:05.934454 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:05.934437    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9451573d-26ec-4478-af0e-7ad34326e92f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9451573d-26ec-4478-af0e-7ad34326e92f" (UID: "9451573d-26ec-4478-af0e-7ad34326e92f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:50:06.033902 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:06.033788    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9451573d-26ec-4478-af0e-7ad34326e92f-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:50:06.033902 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:06.033838    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh9fx\" (UniqueName: \"kubernetes.io/projected/9451573d-26ec-4478-af0e-7ad34326e92f-kube-api-access-dh9fx\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:50:06.293616 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:06.293481    2575 generic.go:356] "Generic (PLEG): container finished" podID="9451573d-26ec-4478-af0e-7ad34326e92f" containerID="1a201dab0e7a539a15cbedd66318e2d864f3a1d1207a7b9c1b02ec4f1c1229b1" exitCode=0
Jul 28 18:50:06.293616 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:06.293575    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"
Jul 28 18:50:06.293859 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:06.293576    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8" event={"ID":"9451573d-26ec-4478-af0e-7ad34326e92f","Type":"ContainerDied","Data":"1a201dab0e7a539a15cbedd66318e2d864f3a1d1207a7b9c1b02ec4f1c1229b1"}
Jul 28 18:50:06.293859 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:06.293737    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8" event={"ID":"9451573d-26ec-4478-af0e-7ad34326e92f","Type":"ContainerDied","Data":"0d7a849980fecc9cdbaab2c18095ac495dcdee1eeb846d5c04bffe30a3c0e6b1"}
Jul 28 18:50:06.293859 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:06.293763    2575 scope.go:117] "RemoveContainer" containerID="1a201dab0e7a539a15cbedd66318e2d864f3a1d1207a7b9c1b02ec4f1c1229b1"
Jul 28 18:50:06.303375 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:06.303351    2575 scope.go:117] "RemoveContainer" containerID="1a201dab0e7a539a15cbedd66318e2d864f3a1d1207a7b9c1b02ec4f1c1229b1"
Jul 28 18:50:06.303709 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:50:06.303684    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a201dab0e7a539a15cbedd66318e2d864f3a1d1207a7b9c1b02ec4f1c1229b1\": container with ID starting with 1a201dab0e7a539a15cbedd66318e2d864f3a1d1207a7b9c1b02ec4f1c1229b1 not found: ID does not exist" containerID="1a201dab0e7a539a15cbedd66318e2d864f3a1d1207a7b9c1b02ec4f1c1229b1"
Jul 28 18:50:06.303775 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:06.303719    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a201dab0e7a539a15cbedd66318e2d864f3a1d1207a7b9c1b02ec4f1c1229b1"} err="failed to get container status \"1a201dab0e7a539a15cbedd66318e2d864f3a1d1207a7b9c1b02ec4f1c1229b1\": rpc error: code = NotFound desc = could not find container \"1a201dab0e7a539a15cbedd66318e2d864f3a1d1207a7b9c1b02ec4f1c1229b1\": container with ID starting with 1a201dab0e7a539a15cbedd66318e2d864f3a1d1207a7b9c1b02ec4f1c1229b1 not found: ID does not exist"
Jul 28 18:50:06.315010 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:06.314973    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"]
Jul 28 18:50:06.318362 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:06.318327    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-dghp8"]
Jul 28 18:50:08.312320 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:50:08.312280    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9451573d-26ec-4478-af0e-7ad34326e92f" path="/var/lib/kubelet/pods/9451573d-26ec-4478-af0e-7ad34326e92f/volumes"
Jul 28 18:51:44.687781 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:51:44.687757    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:51:44.705930 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:51:44.705906    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:54:37.048961 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:37.047804    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"]
Jul 28 18:54:37.774109 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:37.774067    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 18:54:37.782876 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:54:37.782842    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:54:37.782876 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:54:37.782876 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:54:37.782876 ip-10-0-135-188 kubenswrapper[2575]:  > containerID="0e5a2530f7e93540f44970d5242d47d9426e8615db67139c41c8d726121f46ce" cmd=["/bin/sleep","15"]
Jul 28 18:54:37.783156 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:37.782902    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" message="" err=<
Jul 28 18:54:37.783156 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:54:37.783156 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:54:37.783156 ip-10-0-135-188 kubenswrapper[2575]:  >
Jul 28 18:54:37.783156 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:54:37.782919    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:54:37.783156 ip-10-0-135-188 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:54:37.783156 ip-10-0-135-188 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Jul 28 18:54:37.783156 ip-10-0-135-188 kubenswrapper[2575]:  > pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="611db002-424a-4342-a932-c1a4f31b189a" containerName="main" containerID="cri-o://0e5a2530f7e93540f44970d5242d47d9426e8615db67139c41c8d726121f46ce"
Jul 28 18:54:37.783156 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:37.782963    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="611db002-424a-4342-a932-c1a4f31b189a" containerName="main" containerID="cri-o://0e5a2530f7e93540f44970d5242d47d9426e8615db67139c41c8d726121f46ce" gracePeriod=60
Jul 28 18:54:38.024003 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.023978    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:54:38.092248 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.092164    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-dshm\") pod \"611db002-424a-4342-a932-c1a4f31b189a\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") "
Jul 28 18:54:38.092248 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.092216    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-model-cache\") pod \"611db002-424a-4342-a932-c1a4f31b189a\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") "
Jul 28 18:54:38.092248 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.092242    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsrkf\" (UniqueName: \"kubernetes.io/projected/611db002-424a-4342-a932-c1a4f31b189a-kube-api-access-hsrkf\") pod \"611db002-424a-4342-a932-c1a4f31b189a\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") "
Jul 28 18:54:38.092819 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.092278    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-tmp-dir\") pod \"611db002-424a-4342-a932-c1a4f31b189a\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") "
Jul 28 18:54:38.092819 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.092292    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/611db002-424a-4342-a932-c1a4f31b189a-tls-certs\") pod \"611db002-424a-4342-a932-c1a4f31b189a\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") "
Jul 28 18:54:38.092819 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.092329    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-home\") pod \"611db002-424a-4342-a932-c1a4f31b189a\" (UID: \"611db002-424a-4342-a932-c1a4f31b189a\") "
Jul 28 18:54:38.092819 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.092500    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-model-cache" (OuterVolumeSpecName: "model-cache") pod "611db002-424a-4342-a932-c1a4f31b189a" (UID: "611db002-424a-4342-a932-c1a4f31b189a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:54:38.092819 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.092602    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "611db002-424a-4342-a932-c1a4f31b189a" (UID: "611db002-424a-4342-a932-c1a4f31b189a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:54:38.092819 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.092623    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-home" (OuterVolumeSpecName: "home") pod "611db002-424a-4342-a932-c1a4f31b189a" (UID: "611db002-424a-4342-a932-c1a4f31b189a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:54:38.094493 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.094472    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-dshm" (OuterVolumeSpecName: "dshm") pod "611db002-424a-4342-a932-c1a4f31b189a" (UID: "611db002-424a-4342-a932-c1a4f31b189a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:54:38.094576 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.094508    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/611db002-424a-4342-a932-c1a4f31b189a-kube-api-access-hsrkf" (OuterVolumeSpecName: "kube-api-access-hsrkf") pod "611db002-424a-4342-a932-c1a4f31b189a" (UID: "611db002-424a-4342-a932-c1a4f31b189a"). InnerVolumeSpecName "kube-api-access-hsrkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:54:38.094665 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.094636    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/611db002-424a-4342-a932-c1a4f31b189a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "611db002-424a-4342-a932-c1a4f31b189a" (UID: "611db002-424a-4342-a932-c1a4f31b189a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:54:38.193186 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.193147    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-model-cache\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:54:38.193186 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.193179    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsrkf\" (UniqueName: \"kubernetes.io/projected/611db002-424a-4342-a932-c1a4f31b189a-kube-api-access-hsrkf\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:54:38.193410 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.193194    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-tmp-dir\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:54:38.193410 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.193223    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/611db002-424a-4342-a932-c1a4f31b189a-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:54:38.193410 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.193235    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-home\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:54:38.193410 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.193245    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/611db002-424a-4342-a932-c1a4f31b189a-dshm\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:54:38.602376 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.602337    2575 generic.go:356] "Generic (PLEG): container finished" podID="611db002-424a-4342-a932-c1a4f31b189a" containerID="0e5a2530f7e93540f44970d5242d47d9426e8615db67139c41c8d726121f46ce" exitCode=0
Jul 28 18:54:38.602591 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.602390    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:54:38.602591 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.602431    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"611db002-424a-4342-a932-c1a4f31b189a","Type":"ContainerDied","Data":"0e5a2530f7e93540f44970d5242d47d9426e8615db67139c41c8d726121f46ce"}
Jul 28 18:54:38.602591 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.602498    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"611db002-424a-4342-a932-c1a4f31b189a","Type":"ContainerDied","Data":"c1c0999cb2c5b2a65567dd8d12f779e27214fe4e803b1b956d0a97a8f9ba7700"}
Jul 28 18:54:38.602591 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.602547    2575 scope.go:117] "RemoveContainer" containerID="0e5a2530f7e93540f44970d5242d47d9426e8615db67139c41c8d726121f46ce"
Jul 28 18:54:38.611982 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.611963    2575 scope.go:117] "RemoveContainer" containerID="0e5a2530f7e93540f44970d5242d47d9426e8615db67139c41c8d726121f46ce"
Jul 28 18:54:38.612258 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:54:38.612224    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5a2530f7e93540f44970d5242d47d9426e8615db67139c41c8d726121f46ce\": container with ID starting with 0e5a2530f7e93540f44970d5242d47d9426e8615db67139c41c8d726121f46ce not found: ID does not exist" containerID="0e5a2530f7e93540f44970d5242d47d9426e8615db67139c41c8d726121f46ce"
Jul 28 18:54:38.612356 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.612267    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5a2530f7e93540f44970d5242d47d9426e8615db67139c41c8d726121f46ce"} err="failed to get container status \"0e5a2530f7e93540f44970d5242d47d9426e8615db67139c41c8d726121f46ce\": rpc error: code = NotFound desc = could not find container \"0e5a2530f7e93540f44970d5242d47d9426e8615db67139c41c8d726121f46ce\": container with ID starting with 0e5a2530f7e93540f44970d5242d47d9426e8615db67139c41c8d726121f46ce not found: ID does not exist"
Jul 28 18:54:38.631251 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.631224    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 18:54:38.639018 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:38.638995    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 18:54:40.312855 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:40.312808    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="611db002-424a-4342-a932-c1a4f31b189a" path="/var/lib/kubelet/pods/611db002-424a-4342-a932-c1a4f31b189a/volumes"
Jul 28 18:54:45.240655 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:45.240616    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr" podUID="25adce7a-e90e-41ac-9b24-865bc444cae1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:54:52.076282 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.076238    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr" podUID="25adce7a-e90e-41ac-9b24-865bc444cae1" containerName="main" containerID="cri-o://c55a9fae774c93cf6b3283fcf02e2f85d2b9ca5edd0ff8ee54b7524c0e9996fa" gracePeriod=45
Jul 28 18:54:52.338502 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.338441    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"
Jul 28 18:54:52.424835 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.424795    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25adce7a-e90e-41ac-9b24-865bc444cae1-tls-certs\") pod \"25adce7a-e90e-41ac-9b24-865bc444cae1\" (UID: \"25adce7a-e90e-41ac-9b24-865bc444cae1\") "
Jul 28 18:54:52.425042 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.424866    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5rz7\" (UniqueName: \"kubernetes.io/projected/25adce7a-e90e-41ac-9b24-865bc444cae1-kube-api-access-p5rz7\") pod \"25adce7a-e90e-41ac-9b24-865bc444cae1\" (UID: \"25adce7a-e90e-41ac-9b24-865bc444cae1\") "
Jul 28 18:54:52.427034 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.427004    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25adce7a-e90e-41ac-9b24-865bc444cae1-kube-api-access-p5rz7" (OuterVolumeSpecName: "kube-api-access-p5rz7") pod "25adce7a-e90e-41ac-9b24-865bc444cae1" (UID: "25adce7a-e90e-41ac-9b24-865bc444cae1"). InnerVolumeSpecName "kube-api-access-p5rz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:54:52.427034 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.427023    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25adce7a-e90e-41ac-9b24-865bc444cae1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25adce7a-e90e-41ac-9b24-865bc444cae1" (UID: "25adce7a-e90e-41ac-9b24-865bc444cae1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:54:52.525505 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.525471    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5rz7\" (UniqueName: \"kubernetes.io/projected/25adce7a-e90e-41ac-9b24-865bc444cae1-kube-api-access-p5rz7\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:54:52.525505 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.525502    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25adce7a-e90e-41ac-9b24-865bc444cae1-tls-certs\") on node \"ip-10-0-135-188.ec2.internal\" DevicePath \"\""
Jul 28 18:54:52.669994 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.669957    2575 generic.go:356] "Generic (PLEG): container finished" podID="25adce7a-e90e-41ac-9b24-865bc444cae1" containerID="c55a9fae774c93cf6b3283fcf02e2f85d2b9ca5edd0ff8ee54b7524c0e9996fa" exitCode=0
Jul 28 18:54:52.670175 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.670001    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"
Jul 28 18:54:52.670175 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.670040    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr" event={"ID":"25adce7a-e90e-41ac-9b24-865bc444cae1","Type":"ContainerDied","Data":"c55a9fae774c93cf6b3283fcf02e2f85d2b9ca5edd0ff8ee54b7524c0e9996fa"}
Jul 28 18:54:52.670175 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.670098    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr" event={"ID":"25adce7a-e90e-41ac-9b24-865bc444cae1","Type":"ContainerDied","Data":"efef086e82afffe5737f3300939627097a8ea0c576bb343bb9a0ae896ec54c6c"}
Jul 28 18:54:52.670175 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.670129    2575 scope.go:117] "RemoveContainer" containerID="c55a9fae774c93cf6b3283fcf02e2f85d2b9ca5edd0ff8ee54b7524c0e9996fa"
Jul 28 18:54:52.679109 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.679087    2575 scope.go:117] "RemoveContainer" containerID="c55a9fae774c93cf6b3283fcf02e2f85d2b9ca5edd0ff8ee54b7524c0e9996fa"
Jul 28 18:54:52.679351 ip-10-0-135-188 kubenswrapper[2575]: E0728 18:54:52.679335    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c55a9fae774c93cf6b3283fcf02e2f85d2b9ca5edd0ff8ee54b7524c0e9996fa\": container with ID starting with c55a9fae774c93cf6b3283fcf02e2f85d2b9ca5edd0ff8ee54b7524c0e9996fa not found: ID does not exist" containerID="c55a9fae774c93cf6b3283fcf02e2f85d2b9ca5edd0ff8ee54b7524c0e9996fa"
Jul 28 18:54:52.679393 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.679360    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55a9fae774c93cf6b3283fcf02e2f85d2b9ca5edd0ff8ee54b7524c0e9996fa"} err="failed to get container status \"c55a9fae774c93cf6b3283fcf02e2f85d2b9ca5edd0ff8ee54b7524c0e9996fa\": rpc error: code = NotFound desc = could not find container \"c55a9fae774c93cf6b3283fcf02e2f85d2b9ca5edd0ff8ee54b7524c0e9996fa\": container with ID starting with c55a9fae774c93cf6b3283fcf02e2f85d2b9ca5edd0ff8ee54b7524c0e9996fa not found: ID does not exist"
Jul 28 18:54:52.691034 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.691012    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"]
Jul 28 18:54:52.696552 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:52.696532    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-sv5qr"]
Jul 28 18:54:54.314782 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:54:54.314737    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25adce7a-e90e-41ac-9b24-865bc444cae1" path="/var/lib/kubelet/pods/25adce7a-e90e-41ac-9b24-865bc444cae1/volumes"
Jul 28 18:55:28.658225 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:28.658191    2575 ???:1] "http: TLS handshake error from 10.0.140.52:48552: EOF"
Jul 28 18:55:28.784478 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:28.784446    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:29.964239 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:29.964205    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:31.142172 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:31.142142    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:32.286492 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:32.286451    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:33.438956 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:33.438914    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:34.586256 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:34.586221    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:35.732758 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:35.732726    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:36.931137 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:36.931045    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:38.086818 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:38.086773    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:39.247217 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:39.247185    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:40.445819 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:40.445787    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:41.601749 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:41.601715    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:42.757806 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:42.757773    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:43.922690 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:43.922657    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:45.078856 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:45.078828    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:46.234932 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:46.234895    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:47.383912 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:47.383828    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:48.510239 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:48.510202    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:49.632441 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:49.632407    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:50.884958 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:50.884920    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:52.031299 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:52.031267    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:53.195339 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:53.195305    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:54.334775 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:54.334742    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:55.496050 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:55.496016    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:56.630759 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:56.630721    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:57.765336 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:57.765295    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:55:58.902973 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:55:58.902929    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:56:00.036906 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:00.036874    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:56:01.257003 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:01.256970    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:56:02.405865 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:02.405822    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:56:03.563370 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:03.563327    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:56:04.703891 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:04.703849    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:56:05.908650 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:05.908616    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:56:07.050213 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:07.050178    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:56:08.183668 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:08.183624    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:56:09.308201 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:09.308165    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:56:10.441203 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:10.441161    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:56:15.529160 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:15.529123    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8sv5v_2de9f57b-63bf-4dad-be1a-86716a7758f8/global-pull-secret-syncer/0.log"
Jul 28 18:56:15.749557 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:15.749491    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vvdjg_1ab5d559-43a9-4c98-9576-48e592b09677/konnectivity-agent/0.log"
Jul 28 18:56:15.774830 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:15.774796    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-188.ec2.internal_11997b299c0cb2944eaf7c6feeeb8aab/haproxy/0.log"
Jul 28 18:56:20.275913 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:20.275882    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2xg2c_f48667b0-bf40-4193-890e-9acda58c29f3/manager/0.log"
Jul 28 18:56:22.147600 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.147569    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-gd67w_04d98107-270d-46aa-b9e4-013adb971e13/cluster-monitoring-operator/0.log"
Jul 28 18:56:22.174531 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.174481    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-vvvrp_88fd4d31-43e6-41f6-984e-abec4a426779/kube-state-metrics/0.log"
Jul 28 18:56:22.191909 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.191871    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-vvvrp_88fd4d31-43e6-41f6-984e-abec4a426779/kube-rbac-proxy-main/0.log"
Jul 28 18:56:22.211293 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.211265    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-vvvrp_88fd4d31-43e6-41f6-984e-abec4a426779/kube-rbac-proxy-self/0.log"
Jul 28 18:56:22.296560 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.296461    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-66zdn_47211b67-018e-4ffb-9a28-081195f36db3/node-exporter/0.log"
Jul 28 18:56:22.318818 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.318788    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-66zdn_47211b67-018e-4ffb-9a28-081195f36db3/kube-rbac-proxy/0.log"
Jul 28 18:56:22.339868 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.339844    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-66zdn_47211b67-018e-4ffb-9a28-081195f36db3/init-textfile/0.log"
Jul 28 18:56:22.624479 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.624442    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e484d34c-a1f2-4c8c-8df3-5e517676ef79/prometheus/0.log"
Jul 28 18:56:22.646040 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.646014    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e484d34c-a1f2-4c8c-8df3-5e517676ef79/config-reloader/0.log"
Jul 28 18:56:22.669887 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.669860    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e484d34c-a1f2-4c8c-8df3-5e517676ef79/thanos-sidecar/0.log"
Jul 28 18:56:22.690988 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.690947    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e484d34c-a1f2-4c8c-8df3-5e517676ef79/kube-rbac-proxy-web/0.log"
Jul 28 18:56:22.712774 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.712747    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e484d34c-a1f2-4c8c-8df3-5e517676ef79/kube-rbac-proxy/0.log"
Jul 28 18:56:22.733723 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.733695    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e484d34c-a1f2-4c8c-8df3-5e517676ef79/kube-rbac-proxy-thanos/0.log"
Jul 28 18:56:22.754851 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.754828    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e484d34c-a1f2-4c8c-8df3-5e517676ef79/init-config-reloader/0.log"
Jul 28 18:56:22.783873 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.783850    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-ng7mq_74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2/prometheus-operator/0.log"
Jul 28 18:56:22.803119 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.803094    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-ng7mq_74a15adc-a35a-49c4-a2fe-ccf52fbe2ea2/kube-rbac-proxy/0.log"
Jul 28 18:56:22.833402 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:22.833334    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-q6vnx_40036ef4-7b56-419d-83e5-618fdd5121ab/prometheus-operator-admission-webhook/0.log"
Jul 28 18:56:25.249443 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:25.249411    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/1.log"
Jul 28 18:56:25.254873 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:25.254850    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c679z_7b1c7280-90b9-49e0-bd9b-6c91ca7e1ff7/console-operator/2.log"
Jul 28 18:56:25.693380 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:25.693332    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c4fff4cb5-l85lf_0f93b4a6-8381-4aa9-83c1-995cd46ca111/console/0.log"
Jul 28 18:56:26.090295 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.090204    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"]
Jul 28 18:56:26.090657 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.090640    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9451573d-26ec-4478-af0e-7ad34326e92f" containerName="main"
Jul 28 18:56:26.090765 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.090660    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9451573d-26ec-4478-af0e-7ad34326e92f" containerName="main"
Jul 28 18:56:26.090765 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.090686    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="611db002-424a-4342-a932-c1a4f31b189a" containerName="main"
Jul 28 18:56:26.090765 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.090696    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="611db002-424a-4342-a932-c1a4f31b189a" containerName="main"
Jul 28 18:56:26.090765 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.090721    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25adce7a-e90e-41ac-9b24-865bc444cae1" containerName="main"
Jul 28 18:56:26.090765 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.090729    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="25adce7a-e90e-41ac-9b24-865bc444cae1" containerName="main"
Jul 28 18:56:26.091019 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.090808    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9451573d-26ec-4478-af0e-7ad34326e92f" containerName="main"
Jul 28 18:56:26.091019 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.090821    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="611db002-424a-4342-a932-c1a4f31b189a" containerName="main"
Jul 28 18:56:26.091019 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.090837    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="25adce7a-e90e-41ac-9b24-865bc444cae1" containerName="main"
Jul 28 18:56:26.094140 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.094116    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.096350 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.096327    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ct488\"/\"openshift-service-ca.crt\""
Jul 28 18:56:26.097071 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.097053    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ct488\"/\"default-dockercfg-gl4lk\""
Jul 28 18:56:26.097071 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.097065    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ct488\"/\"kube-root-ca.crt\""
Jul 28 18:56:26.104820 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.104794    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"]
Jul 28 18:56:26.190571 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.190515    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sn6l\" (UniqueName: \"kubernetes.io/projected/03694051-3ff4-44b4-a3cc-0ac601b3a75c-kube-api-access-5sn6l\") pod \"perf-node-gather-daemonset-v4srt\" (UID: \"03694051-3ff4-44b4-a3cc-0ac601b3a75c\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.190571 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.190576    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03694051-3ff4-44b4-a3cc-0ac601b3a75c-lib-modules\") pod \"perf-node-gather-daemonset-v4srt\" (UID: \"03694051-3ff4-44b4-a3cc-0ac601b3a75c\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.190794 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.190625    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/03694051-3ff4-44b4-a3cc-0ac601b3a75c-podres\") pod \"perf-node-gather-daemonset-v4srt\" (UID: \"03694051-3ff4-44b4-a3cc-0ac601b3a75c\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.190794 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.190711    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/03694051-3ff4-44b4-a3cc-0ac601b3a75c-proc\") pod \"perf-node-gather-daemonset-v4srt\" (UID: \"03694051-3ff4-44b4-a3cc-0ac601b3a75c\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.190794 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.190761    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03694051-3ff4-44b4-a3cc-0ac601b3a75c-sys\") pod \"perf-node-gather-daemonset-v4srt\" (UID: \"03694051-3ff4-44b4-a3cc-0ac601b3a75c\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.291324 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.291289    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/03694051-3ff4-44b4-a3cc-0ac601b3a75c-podres\") pod \"perf-node-gather-daemonset-v4srt\" (UID: \"03694051-3ff4-44b4-a3cc-0ac601b3a75c\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.291771 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.291340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/03694051-3ff4-44b4-a3cc-0ac601b3a75c-proc\") pod \"perf-node-gather-daemonset-v4srt\" (UID: \"03694051-3ff4-44b4-a3cc-0ac601b3a75c\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.291771 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.291361    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03694051-3ff4-44b4-a3cc-0ac601b3a75c-sys\") pod \"perf-node-gather-daemonset-v4srt\" (UID: \"03694051-3ff4-44b4-a3cc-0ac601b3a75c\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.291771 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.291391    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sn6l\" (UniqueName: \"kubernetes.io/projected/03694051-3ff4-44b4-a3cc-0ac601b3a75c-kube-api-access-5sn6l\") pod \"perf-node-gather-daemonset-v4srt\" (UID: \"03694051-3ff4-44b4-a3cc-0ac601b3a75c\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.291771 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.291414    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03694051-3ff4-44b4-a3cc-0ac601b3a75c-lib-modules\") pod \"perf-node-gather-daemonset-v4srt\" (UID: \"03694051-3ff4-44b4-a3cc-0ac601b3a75c\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.291771 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.291449    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/03694051-3ff4-44b4-a3cc-0ac601b3a75c-proc\") pod \"perf-node-gather-daemonset-v4srt\" (UID: \"03694051-3ff4-44b4-a3cc-0ac601b3a75c\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.291771 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.291473    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/03694051-3ff4-44b4-a3cc-0ac601b3a75c-podres\") pod \"perf-node-gather-daemonset-v4srt\" (UID: \"03694051-3ff4-44b4-a3cc-0ac601b3a75c\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.291771 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.291501    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03694051-3ff4-44b4-a3cc-0ac601b3a75c-sys\") pod \"perf-node-gather-daemonset-v4srt\" (UID: \"03694051-3ff4-44b4-a3cc-0ac601b3a75c\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.291771 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.291635    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03694051-3ff4-44b4-a3cc-0ac601b3a75c-lib-modules\") pod \"perf-node-gather-daemonset-v4srt\" (UID: \"03694051-3ff4-44b4-a3cc-0ac601b3a75c\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.299882 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.299855    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sn6l\" (UniqueName: \"kubernetes.io/projected/03694051-3ff4-44b4-a3cc-0ac601b3a75c-kube-api-access-5sn6l\") pod \"perf-node-gather-daemonset-v4srt\" (UID: \"03694051-3ff4-44b4-a3cc-0ac601b3a75c\") " pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.374419 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.374337    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-fsv8l_e9a31f62-4a8d-48c8-8e8a-73079127d4d1/volume-data-source-validator/0.log"
Jul 28 18:56:26.406336 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.406297    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:26.539732 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:26.539706    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"]
Jul 28 18:56:26.542271 ip-10-0-135-188 kubenswrapper[2575]: W0728 18:56:26.542235    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod03694051_3ff4_44b4_a3cc_0ac601b3a75c.slice/crio-6750016d9225b94cfbb1972e37e09e5a6562fb37721104da92473509770e8d8f WatchSource:0}: Error finding container 6750016d9225b94cfbb1972e37e09e5a6562fb37721104da92473509770e8d8f: Status 404 returned error can't find the container with id 6750016d9225b94cfbb1972e37e09e5a6562fb37721104da92473509770e8d8f
Jul 28 18:56:27.111583 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:27.111546    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt" event={"ID":"03694051-3ff4-44b4-a3cc-0ac601b3a75c","Type":"ContainerStarted","Data":"d78b3dc44f430217685258cfa627e80b5aa6c796e6220defb9c34806c1009c51"}
Jul 28 18:56:27.111798 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:27.111606    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt" event={"ID":"03694051-3ff4-44b4-a3cc-0ac601b3a75c","Type":"ContainerStarted","Data":"6750016d9225b94cfbb1972e37e09e5a6562fb37721104da92473509770e8d8f"}
Jul 28 18:56:27.111798 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:27.111637    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:27.129938 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:27.129878    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt" podStartSLOduration=1.129865411 podStartE2EDuration="1.129865411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:56:27.129308863 +0000 UTC m=+5983.386918556" watchObservedRunningTime="2026-07-28 18:56:27.129865411 +0000 UTC m=+5983.387475100"
Jul 28 18:56:27.433973 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:27.433937    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2nw7c_56add391-4f0b-4bf8-a857-563b82d92fbf/dns/0.log"
Jul 28 18:56:27.463297 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:27.463270    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2nw7c_56add391-4f0b-4bf8-a857-563b82d92fbf/kube-rbac-proxy/0.log"
Jul 28 18:56:27.693897 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:27.693806    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gcwqn_bac64312-87a3-4c2c-8cfa-80754d0123ed/dns-node-resolver/0.log"
Jul 28 18:56:28.357504 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:28.357471    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wptfc_f4350004-c24b-49f4-9142-b7d268dd1132/node-ca/0.log"
Jul 28 18:56:30.474434 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:30.474386    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5xnh4_8bff935c-a6ff-41ec-8b79-8146ede38416/serve-healthcheck-canary/0.log"
Jul 28 18:56:30.990469 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:30.990440    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-vkqmg_958ca081-580a-4d51-af0a-093e902e405e/insights-operator/0.log"
Jul 28 18:56:30.992169 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:30.992150    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-vkqmg_958ca081-580a-4d51-af0a-093e902e405e/insights-operator/1.log"
Jul 28 18:56:31.140953 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:31.140923    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zp6vz_269dc00a-7bd4-4a18-97a3-2a1ec916dd93/kube-rbac-proxy/0.log"
Jul 28 18:56:31.160926 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:31.160901    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zp6vz_269dc00a-7bd4-4a18-97a3-2a1ec916dd93/exporter/0.log"
Jul 28 18:56:31.182260 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:31.182238    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zp6vz_269dc00a-7bd4-4a18-97a3-2a1ec916dd93/extractor/0.log"
Jul 28 18:56:33.126019 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:33.125991    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ct488/perf-node-gather-daemonset-v4srt"
Jul 28 18:56:34.093503 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:34.093472    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8994644bb-j4rkm_8a13ad18-f3e8-42f8-8294-a5521deee97a/manager/0.log"
Jul 28 18:56:34.116757 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:34.116723    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-9978k_c96dddf3-5664-4d4c-8ea5-564e8ce807b6/openshift-lws-operator/0.log"
Jul 28 18:56:34.958905 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:34.958875    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-phr94_6df280f9-a317-4ff5-a263-06c154c30f4b/server/0.log"
Jul 28 18:56:35.482448 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:35.482413    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-zxn8g_c9516887-ab55-4bdd-8f42-4a4f5404a174/manager/0.log"
Jul 28 18:56:35.583335 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:35.583305    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-msmzj_68c368f9-042f-4db6-99ad-22d0acd0721a/seaweedfs/0.log"
Jul 28 18:56:42.743040 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:42.743006    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xnpnk_325554ac-10f4-4076-b562-20b27ddf18b0/kube-multus-additional-cni-plugins/0.log"
Jul 28 18:56:42.815552 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:42.815511    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xnpnk_325554ac-10f4-4076-b562-20b27ddf18b0/egress-router-binary-copy/0.log"
Jul 28 18:56:42.846021 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:42.845986    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xnpnk_325554ac-10f4-4076-b562-20b27ddf18b0/cni-plugins/0.log"
Jul 28 18:56:42.871929 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:42.871903    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xnpnk_325554ac-10f4-4076-b562-20b27ddf18b0/bond-cni-plugin/0.log"
Jul 28 18:56:42.894580 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:42.894555    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xnpnk_325554ac-10f4-4076-b562-20b27ddf18b0/routeoverride-cni/0.log"
Jul 28 18:56:42.916734 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:42.916709    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xnpnk_325554ac-10f4-4076-b562-20b27ddf18b0/whereabouts-cni-bincopy/0.log"
Jul 28 18:56:42.939648 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:42.939621    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xnpnk_325554ac-10f4-4076-b562-20b27ddf18b0/whereabouts-cni/0.log"
Jul 28 18:56:43.013609 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:43.013510    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l4s6b_beefa577-daaa-474f-9acf-80700f353213/kube-multus/0.log"
Jul 28 18:56:43.168815 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:43.168778    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zqjzm_0ee703af-bd7c-45e1-8b8a-1b31645d40bf/network-metrics-daemon/0.log"
Jul 28 18:56:43.187047 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:43.187023    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zqjzm_0ee703af-bd7c-45e1-8b8a-1b31645d40bf/kube-rbac-proxy/0.log"
Jul 28 18:56:44.008474 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:44.008440    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c954x_f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe/ovn-controller/0.log"
Jul 28 18:56:44.024068 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:44.024047    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c954x_f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe/ovn-acl-logging/0.log"
Jul 28 18:56:44.047496 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:44.047470    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c954x_f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe/kube-rbac-proxy-node/0.log"
Jul 28 18:56:44.067334 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:44.067299    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c954x_f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 18:56:44.087176 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:44.087122    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c954x_f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe/northd/0.log"
Jul 28 18:56:44.107573 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:44.107549    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c954x_f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe/nbdb/0.log"
Jul 28 18:56:44.130443 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:44.130423    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c954x_f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe/sbdb/0.log"
Jul 28 18:56:44.193408 ip-10-0-135-188 kubenswrapper[2575]: I0728 18:56:44.193380    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c954x_f8e3a41d-925d-44b9-a6cb-ccf17cabd6fe/ovnkube-controller/0.log"